Yuval Adam

The Anomaly of Amazon EC2 Load Balancing

Amazon has a cool load balancing service as part of their EC2 offering. Sure, you can always set up a load balancer of your own (HAProxy is my favorite), but if you’re already running on EC2, using the Amazon load balancer is an easy option. So we went with that.

Until we discovered a weird issue, that has already been a problem for quite some time. The problem is actually very simple - when starting an Amazon LB, it is automatically assigned an Amazon public DNS address. Cool. After configuring it, all you need to do is just point a CNAME DNS record to it, and you’re done.

But! What if you want to load balance your web servers? What if you want to serve your website from the root domain (example.com)? How do you set a CNAME record for the root domain? Simple: you can’t. Even if you wanted to create an A record and point it to the load balancers public IP, you’re in no luck - Amazon still does not allow attaching a public IP to a load balancer. (This blog post has an in-depth explanation of why this is the case.)

We found ourselves in this situation today, with the following options:

  1. Set the CNAME record from the www subdomain and serve all web content from www.example.com. We did not like this option since this meant changing our policy of serving our site from example.com. It also meant having a nasty redirect from example.com to www.example.com to anyone that accesses links without the subdomain.
  2. Set up a proxy server just for accepting incoming connections, and forwarding them to the load balancer.

We went with the proxy server solution. It’s ugly. It’s a hack. It adds network latency. But until Amazon add the option to attach an IP to a load balancer, it’s the best option we have. We set up a micro instance, installed HAProxy (with the following configuration), and it works pretty good.

So, Amazon, how about those public IPs?

Fabric SSH With Keyfile

Probably due to Fabric’s awkward SSH support (via Paramiko), I could not - for the life of me - find a simple example of a simple remote command execution via SSH with keyfile authentication.

If you’re a newbie to Fabric, you’ll probably find this a good start. Enjoy.

Startup Weekend TLV 2011

TL;DR; In Israel - ideas are valued ridiculously higher than execution. All talented developers are, most likely, enjoying a decent salary and a normal life at their corporate jobs.

I just got back from the latest Startup Weekend that took place in Tel Aviv, Israel. This was my second time attending, and as a hardcore web developer, I was looking forward to an action-packed hackathon. That wasn’t the case. If at the first SW I was busy soaking up the atmosphere, this time I was more perceptive to understanding and analyzing what’s going on around me.

Now, don’t get me wrong, I had a great time. My idea was one of the 10 chosen, and I had the pleasure of working with a wonderful group of people. Not only that, the convention was organized very professionally, big props go to Eddy, Moran and Josh for the great work!

However, I do have my gripes with Startup Weekend; and I feel that if SW - even remotely - resembles the situation of the entire entrepreneurial community in Israel, then we have ourselves a problem.

From the first moment, I was looking around to meet not only familiar faces (which I was happy to see quite a few of), but more importantly to get acquainted with new, talented and eager developers.

First up - idea pitches. The majority of the pitches were very badly presented. I can’t blame anyone for having a lame idea, since there is no such thing in my book - almost all ideas are good enough that you can build a prototype on their premise, and validate against a certain market. What I do care about is half-baked ideas that are presented badly. Half of the presenters didn’t even know what and who they needed in order to execute their idea!

OK, so not everyone knows how to build a team and a product. Fair enough.

When we got to the recruiting stage, just as I stated in my pitch, I was looking out for the best and brightest web developers I could find in the crowd. I was aiming for a full blown hackathon. Looks like I had another thing coming. My team consisted of 3 developers besides me, and 7 (!!) biz devs. Other teams were smaller, but had a similar ratio of talented developers to others. Designers were nowhere to be seen.

If I was aiming to split my time between a development team and a product team - it was clear to me now that I had to invest 150% of my time in hacking together with the developers. Which would be awesome otherwise, but left me very little time to bounce ideas back and forth with the biz devs. We had two very strong front-end developers, but the back-end dev unfortunately had to bail early into the second day. So that left me tackling the server side by myself.

My biz dev team was very diverse, and consisted of people with a vast array of talents and experience. They had lots of great ideas, but since none of the devs had much free time - we just didn’t get to exchanging ideas.

That put my team in a very bad position. However, we did manage to meet my goal - which was to reach demo day with a working prototype that would have some “wow” effect on the crowd. (We had several bugs I wish we could solve, but given the circumstances, I think we did pretty good.) The problem for me was that I did not have any time left to work on the presentation with the biz devs. I very much wish I could incorporate more of their ideas into the final version.

At the end - the presentation went pretty good, and the demo was up and running. But it could have been much better.

I must say that most of the other teams did, eventually, focus more on the P-def side of things, and less on building an impressive demo. All the other teams had great ideas, but it was clear that the vast majority did not focus on getting a demo ready.

I don’t know if this is the result of correlation or causation - but one thing is clear to me. Ideas are valued much, much higher than execution in the local entrepreneurial community. All along the event I was asking myself “where are all the hackers?”. This can’t be it.

There is no way that Israel, with all the amazing things that are going on in academia and in the military (e.g. Unit 8200), cannot created a generation of talented hackers that can deliver cutting edge technological products.

In order for the next Startup Weekend to be successful, in my eyes, we must be able to bring more talents that can get shit done. Bouncing ideas back and forth will only get you so far.

Paul Graham, which I respect a great deal, said that working from a startup hub is paramount. At first I thought he’s talking nonsense.

Now, I think I am starting to get it.