This weekend I attended Rockaway AWS Hackathon. The topic of the event were technologies of Amazon Web Services. We use some of them in our company to run ABRA FlexiBee. But I wanted to see more of them. I was born megalomaniac so I was interested in technologies that will allow me to scale my applications to the sky.

The hackathon consisted of two parts - presentations and mainly programming. Programming of a chosen task in small teams during 30 hours. The organizers wanted us to really touch the technology. So they prepared credits in total amount of ca. 8 000 $ (ca. 200 000 CZK). And that’s a lot for a 24-hour meet-up. For better imagination - it is large enough to allow us run FlexiBee for a half of a year. So I said to myself - I want my application so scalable and massive that I will spend all this money alone :-)

One of the most interesting presentation there was AWS Lambda by Štěpán Bechynský. AWS Lambda is a small piece of a program that you can upload to AWS. And it’s easy to run it. It can be quiet large, but the time spend during invocation has to be within chosen limit (usually between 5 seconds to 5 minutes maximally). You just write the program and the rest is handled by Amazon.

Later there were presentations of ideas and split into teams. Nearly every other attendant already knew what they want to do ahead. But I wasn’t preparing for this hackathon in advance, so I had neither team nor idea. I thought I will do it on place. So I became a member of popular team “Miserables”. Those who remained at each others and didn’t know what they will do.

We were discussing for a while that we will make analysis of transparent bank accounts using WebAPI from Česká Spořitelna. But at the end we chosed idea of one of the organizers (my wife later told that it was smart step :-).

Our application

We decided to make simple application to track sharing of web page address (URL) on social medias (Facebook, Twitter, Google+ and LinkedIn). User enters web address and service shows him number of sharing. Then it checks actual state every 5 minutes. That is stored and user can see history of it.

Result can be seen on this video. Screenshots are at the end of this article:

The Team

There were three of us in our team: young talented graphic designer Marek Dlugoš, programmer Lukáš Kříž and me. With a modesty of my own, I took over the leadership of the team. Marek gave our application the most important thing and that was a nice design. Lukáš and I programmed everything.

The Application

We built the application so that it was a set of AWS Lambda tasks in node.js, which were called using the AWS API Gateway. Individual tasks are called from different servers and IP addresses, which relieved us of worries about the limits on the number of API calls to social networking services. We first wrote the basics of the application and then began to address its ability to scale. We had 18,000 different addresses checked and tested the load. Of course, unsuccessfully. The problem was just the maximum timeout of the running application. We loaded the processing links from the database and started to divide the tasks. But it took a long time. We tried a lot of dead ends until we finished loading all the addresses, dividing them into hundreds, sending them to another service, which ensured the start of subtasks with checking individual addresses. In about 20 seconds we were able to test tens of thousands of addresses at once. In total, we performed one million inspections in two hours. That means we’ve made a million requests on Facebook, a million on Twitter, a million on LinkedIn, and a million on Google+. I found the change of thinking very interesting, where instead of counting an hour with one computer, I give the task to tens of thousands of computers and I have the result in 10 seconds. Contrary to reality, AWS has the idea that 9 women carry a baby a month. When we talked to the organizers about our application during the implementation, they politely indicated to us that we should not consume all the prepared credit ourselves. We are probably not understanding :-) In the end, the limiting performance of the SQL database, which we used to store the results, proved to be more scalable. Each task turned directly to her and had to handle a huge number of queries at once. Only one server took care of the database operation. Although very powerful, but it simply could not be enough to handle traffic from thousands of servers. Next time, we would not let the final tasks communicate directly with the database, but will only return the retrieved data to the parent, who will store the data in summary. After editing, we could process a thousand times more tasks in the same time.

How we spend other’s money

When our application finally started doing what it was supposed to do, it started performing hundreds of thousands of operations. I imagined how the allotted credit disappeared like steam over a pot and how everyone else would be soapy that their application would have nothing to run on. I have already seen our team as the winner of the biggest spend. So I went to check it out. I sat and rejoiced mischievously. But then I had the cruel thought of whether our service is connected to the credit of the organizers. A moment of tension. Was not! How come, there’s my own credit card! I was drenched in cold sweat. I quickly ran after the organizers to connect our application to their account. I went to see how much the fun cost me. In my mind, I was preparing for personal bankruptcy. Fortunately, I almost didn’t exceed the credit that AWS offers to developers for free and it only cost me $ 6 :-)

Presentation

I was already terribly tired and sleepless at the presentations. When the organizers unexpectedly asked me what we were actually doing, I couldn’t say a word. That doesn’t happen to me very often ;-)

And how did we end up?

We were the second first :-). The organizers placed great emphasis on the fact that we are not the second, but that the two are the first. They marked one as the first because they had only one main prize. We got a bass of beer and Marek took it home. He wrote that he handed out most of them in the subway because it was too heavy :-) Personally, I consider Mark the winner of the whole event, because he added design to both winning teams and maybe arranged a new job right there. On [his story] (http://marekdlugos.github.io/TakeMeAsIntern/) we also built a defense of the whole application. I want to thank the organizers for a successful event. And if you are interested in the event, you can also take part in it in Brno and Ostrava. The feeds are available on [GitHub] (https://github.com/fersman/share-analytics.io “> GitHub . Due to lack of time, they are quite hacked and you can’t infer anything from them :-)