Python Google Trends API

google

EDIT: FYI PEOPLE —> I RETIRED THE REPO AT BIT BUCKET, not to panic though, dreyco676 has a version for python 3 and it is working well as of 21/10/2014. Please see https://github.com/dreyco676/pytrends.

Hi once again, slight detour but thought I should share this. I found the original code on-line and with a few tweaks, managed to get it to do what I wanted and so here it is for anyone interested in this sort of thing. In a world where large datasets are becoming ever more available to the average Joe, Google are doing their bit by allowing you to see what historic rates of search terms have occurred over a given time period, essentially allowing one to look back at what people have been thinking about. You can try this for yourself by following this link to http://www.google.co.uk/trends homepage. As an example lets say I was curious to see how often people search for the term fruit, I would get the following display.

googletrend

(ARGH, since this blog is free I cannot embed the link!!) There are plenty of ways one can then analyses these forms of data whether it be sentiment indicators such as the stock market, movie hits based on search results, when are most babies born and other such seasonal traffic patterns just as some examples.

Anyway the code is simple python script and follows a simple example so check it out. It does require you to login to your Google account so that it can cache the cookies so if you don’t have one you can always get each “.csv” file you need directly from the Google Trends website posted above. EDIT 20/01/2014: You will not be able to login if you have Google additional security running, this is because you get a redirect java session that will wait for a password that is sent to you by mobile, thus the script will never know what that is and is not written in a way to accept it as input. Don’t turn your security off to use this script, thats just stupid, instead just try/make a different gmail account, sorted!!

# Download the git repo.
git clone https://github.com/dreyco676/pytrends.git
cd pytrends
./example.py

EDIT: –> As mentioned above please use the pytrends version on github. I will not be supporting mine. There is an example called example.py, run this and you sould download a search for “pizza”!

This simple example will go and grab a load of trend data provided in the python list and store each in a .csv file. You can also check out the repository directly at https://bitbucket.org/mattreid9956/google-trend-api. The formatting is such that you are returned the end of week date for the whole week and the trend value over that period, this is supposed to make life easier should one run an analysis later. One could easily modify this script to get the desired formatting, note that period in which you search will change the granularity of the time window. For instance searches for 3 months will return daily results, where as searches over a year will return the accumulated results over a given week. This is a little annoying an I don’t see why Google won’t allow daily results by default, maybe time to ask them! Watch this space…

Boost.Python. Executing C++ inside a python environment

CppPython
Python is a fantastic scripting language having a simple and easy to learn syntax and style. At LHCb we use Python as a configurable to control the desired functionality that is usually embedded in C++. So why would you want to have two languages? Does that now make things more complicated? Well C++ is inherently much faster to operate so when we have to make decisions which need to be calculated within a few milliseconds we utilise the speed of C++. However, if you have a bunch of classes that are written and just need executing it can be much easier to run a python script to control and setup a job that we want to run as it is a much simpler scripting language. What usually takes many lines of code to do in C++ can be achieved in just a few lines of python. So, I wanted to show an example of a C++ class that can be used as an object in a python instance. The example follows on from the simple vector class (x,y,z) used in the tutorials, https://particlephysicsandcode.wordpress.com/learn-c-and-special-relativity. With this example we will see how Boost provides tools that converts your class into a fully fledged python object. This means an identical class can be used between platforms, python or C++. To get the source, wget the following:-

wget https://dl.dropbox.com/u/88131281/boost_python_example.tar.gz
tar xzvf boost_python_example.tar.gz
cd boost_python_example
chmod +x build.sh
./build.sh

I assume you have cmake, if not a simple “sudo apt-get install cmake” should do the trick. In the build directory created there will be a folder called Vector. The clever part is all done by the Boost.Python template, the one we use in this example is BOOST_PYTHON_MODULE(vectors). Using this wrapper class you can add all the functionality that your class exhibits then behind the scenes the C++ compiler converts this to C-binary understood by python. Running the python script you should see the following:-

matt@matt-W250ENQ-W270ENQ:~/C++/boost_python_example$ ./build/Vectors/vectors.py 
vec  = (1, -3.56019, 0.570154)
vec2 = (4.44649, 3, 0.478267)
vec3 = (5.44649, -0.560186, 1.04842)
(1, 1, 1)
(2, 5, 7)
8.83176086633
(R, Phi, Theta) = (8.83176086633, 1.19028994968, 0.655744935261)

These are a series of example just to show that the implementation is correct and gives the desired results, we can rotate about the axes, get the length of the vector, change to polar coordinates… Everything we can do in the C++ class. For this vector example we define the following forward declarations of our class:-

//...
// Add the required python headers
#include <boost/python.hpp>
#include <boost/python/operators.hpp>

//...
// The main class definitions
// ...

// Add the python module
using namespace boost::python;
BOOST_PYTHON_MODULE(vectors)
{
    boost::python::class_<ThreeVector>("ThreeVector")
        .def( init<ThreeVector>() )
        .def( init<double, double, double>() )
        .def(self_ns::str(self_ns::self))
        .def( self + ThreeVector())
        .def( self - ThreeVector())
        .def( self * ThreeVector())
        .def( self / ThreeVector())
        .def("setXYZ", &ThreeVector::setXYZ)
        .add_property("X", &ThreeVector::getX, &ThreeVector::setX)
        .add_property("Y", &ThreeVector::getY, &ThreeVector::setY)
        .add_property("Z", &ThreeVector::getZ, &ThreeVector::setZ)
        .add_property("R", &ThreeVector::getR, &ThreeVector::setR)
        .add_property("Theta", &ThreeVector::getTheta, &ThreeVector::setTheta)
        .add_property("Phi", &ThreeVector::getPhi, &ThreeVector::setPhi)
        .def("length", &ThreeVector::length)
        .def("rotateX", &ThreeVector::rotateX)
        .def("rotateY", &ThreeVector::rotateY)
        .def("rotateZ", &ThreeVector::rotateZ);

    boost::python::def("arctan", arctan);
    boost::python::def("scalarProduct", scalarProduct);
};

Firstly we declare the class “ThreeVector”. Then we declare the constructors with the init wrapper, “.def( init() )” followed by the copy assignment operator “.def(self_ns::str(self_ns::self))”. To define operators we use self along with the operator as written above “.def( self + ThreeVector())”. To access getter and setter functions we use the “.add_property(“Name”, &MyClass::getName, &MyClass::setName)”. Any remaining public member functions of the class can be added with the “.def(“func”, &MyClass::func)”. Anyway, that about sums it up for a class. If you have free functions that you want to forward declare for use in python use the def function as shown, “def(“arctan”, arctan);”. It is important to remember the semi-colon at then end of the module declaration and to use full stops after each class attribute that you specify.

Interpolation of a C++ class to a python one is simple with Boost and can be useful for anyone wishing to expand the usability of their code so that python programmers can use it, or simply allow one to write very quick programmes for testing. You can simply open a python terminal import the Vectors object and you’re in business.