Under the hood of my Hack24 entry

I competed by myself at hack24. I don’t see competing alone as a massive issue personally, as it cuts down communication overhead by a massive amount. 😉

For those that haven’t heard of hackathons before, the concept is fairly simple:

  • Teams of 1-4 people.
  • Multiple challenges (at least in this event)
  • 24 hours to build a solution.
  • Only code you have written during the event (or Open Source libraries/apps) can be used.

It is quite gruelling to compete in these types of events; you have limited time and you need to prioritise your time across features and components carefully so you don’t end up with a complete backend but no frontend or similar.

Generally you omit anything that is not-essential such as login systems, logging or some error checks etc.

I used the event to test some of the ideas and applications (Microservices, RabbitMQ, Riak, Nginx, php-fpm etc) I have been using on larger projects in a smaller time constrained project – and I think the outcome proves that it works.

I am a big fan of microservices; breaking your app into small testable components with fixed interfaces is awesome for quick building and easy debugging – want to test if something is working, inject a message into the input and see what the outputs are. Once working you can mostly treat it as a single component and move onto the next piece. It keeps your mind focused and, at least for me, reduces stress.

I am a long-time PHP coder and so it made sense, when it came to quick prototyping, to use this and the lightweight Slim framework as my language of choice – both for the frontend and all the workers on the backend all running on top of a single Vultr VPS [disclosure: url contains affiliate link].


SMS proxy – my hack24 entry

I broke the project into 4 main parts:

  • Frontend (website, sms callback hook for receiving messages from Esendex (the challenge sponsor))
  • Receiving worker (take messages from callback and lookup entries in Riak and route to next processing step either filter or directly to sending worker)
  • Filter (responsible for transforming the message before routing the message to the sending worker)
  • Sending worker (responsible for sending the message out via Esendex)

I am open-sourcing the code as a learning tool for other people (and perhaps so I can abuse it in later events…)

You are welcome to ask any questions and I will do my best to answer them.

If you want to contact me, please use twitter or email mike @ technomonk . com

Microservices payload design

There are a lot of people talking about microservices at present. I understand it is fashionable and a lot of people are trying to get rich from consulting in the domain, but a lot of the things I hear are just plain wrong or bad practice.

People are just throwing microservices out there with little consideration of any of the basic requirements of operations. No idea of how they will monitor them; no concept of of how they can debug them and no regard as to how they upgrade them.

I came from a dev background and over time migrated into operations. I dislike fragile, hard to debug components.

I have learnt the hard way just how difficult a distributed microservice architecture can be – but it is still (when designed correctly) a better choice than a hulking monolith of code.

I like my payloads to be debuggable. For this reason I like JSON as a interchange format. It isn’t the most efficient representation format but can be easily human-parsed and is supported by most languages.

I like to know how long a request has taken through the system so I put a uuid and microsecond timestamp when every request is first seen. I also like to know how long each step takes so I also append a timestamp and additional id for every process the request passes through.

Doing these in a uniform way allows you to take a payload from anywhere in the system and be able to use the same tools to perform basic analysis.

It can bloat the payload a little, but being able to see at a glance how quickly each worker is dealing with requests, how long each request takes end to end and what bottlenecks you are seeing allows you to define and keep contracts with your service consumers and detect potential failures before they happen and automatically trip circuit breakers to mitigate some of these issues.

I don’t have all the answers, but this is starting to work well for me.

Just say no

Some time back I became aware that I said “Yes” to too many things. I liked to please people and taking on additional burdens didn’t seem like it was much of a problem.

If I was great in managing my time and nothing outside of my control came to upset the balance I would have been fine, but I wasn’t. I was in some ways as far from great time management as the North pole is from the South – literally poles apart.

Like any habit, it only becomes so if you actually do it. I have made it my goal to say no and mean it at least once a day when people ask me for stuff.

So far it is definitely helping. I have less things on my task-list and I seem to be making a dent in some things I haven’t even looked at for months. Once you start seeing progress you start to feel happier about it and it starts to become a positive feedback loop – Bonus!

I just hope I can keep it up for the next 40 days or so, so it becomes an automatic habit.

What are you waiting for?

Are you waiting on a lightening strike?
Are you waiting for the perfect night?
Are you waiting till the time is right?
What are you waiting for?

— Nickleback, “What are you waiting for”

It seems that a lot of people are waiting for the perfect moment, and up until recently I too used it as an excuse.

That is all it is; an excuse. The time will never be perfect in your mind. There will always be something that is sub-optimal, so get off your arse and just do it. Use what resources you have right now to do the best you can and improve on it later, but just get something out there.

I have literally wasted years by not following through on ideas. Many of which I have seen other people succeed with just a year or two after I didn’t even bother to try. If I had done something rather than clicking through to the next funny cat picture, like a rat pushing a bar for the next food pellet, I would certainly be in a better position than today. I might not have succeeded, but I would have at least tried and whether I failed or succeeded I would have got the experience that only trying provides.

Quick, good or cheap – choose two

I often go to start-up events and I am still suprised how few people even bother to do even basic sanity checking about their business idea.

Either something like:

“I want to build the next Facebook”
“How much budget do you have?”
“About £500 and that includes marketing”

Where they have no idea about manpower or infrastructure costs, how they will monetise it or anything else…

“I want to build a site where people can get little fixed-price jobs done”
“You mean like fiverr.com?”
“You mean there is already a site like that?”

Never even bothering to do basic research…

“I want to build a site where people can buy our [dog] products, but I need it completely writing from scratch”
“Our software should be what makes us unique!”
“So you want to sell this e-commerce platform to other people?”
“Nope, it will be ours and ours alone”
“Why don’t you customise an open-source platform?”
“Because other people can copy us easier”

Where they just don’t understand why they are asking for what they are asking for.

The main thing most of these people have in common is that they are not playing to their strengths. Their strength is certainly not software development or website design but they think they know enough to manage a project in these fields. Unfortunately, most of these people end up with developers that know little more than they do and a project doomed to fail.

They have failed to even do the basics of validating that the market exists and who the existing players are, what problem they are trying to solve and if they have defined requirements (like writing your code from scratch) why this requirement even exists.

For me the thing that saddens me the most is that many of these people have sunk weeks and weeks of evenings and weekends, planning what they want the site to look like without even bothering to lay the groundwork; frequently not understanding that most of what they have designed is just composed of basic design patterns.

Spending an hour with an expert (me or anyone else) could have saved them a lot of wasted time and in some cases money.

If you do feel that an hour or paid time with myself would help you get your ideas straight, then you can contact me on mike-at-technomonk-dot-com (replacing the necessary parts) or +44(0)7950892038 / skype:darkflib


The phrase in the title comes from an old idea in that you can only have 2 of the 3:

  • A good and cheap implementation wont be quick.
  • A cheap and quick implementation wont be good.
  • A good and quick implementation wont be cheap.

There is some truth in it, but with the advent of frameworks and open source platforms like WordPress and Drupal, it becomes less so.

Instant Messaging

I was asked after my last post what I used for instant messaging. The answer is kinda interesting in my opinion.

As some of you know I am a privacy advocate, but try to also balance that with ease of use and ease of integrating with other people.

I run my own XMPP server which I share with a number of collegues. This is running of Prosody an open source XMPP server. We use the MUC module to host a number of conference rooms where our bots and other notifications are sent.

In addition to this I also use Skype, not because I trust Microsoft to not sell my contact info out to the NSA, but because it reduces the friction of contacting me for a number of people. Forcing them to use my private XMPP server (even through federation) is too big a hurdle for them to jump.

Email overload

Ask anyone that has been online with the same email address for more than a few years and you’ll rarely hear them say that they have no issue with spam.

The truth is that as soon you start to use an email address for anything like everyday use, you will start to accumulate spam and other unwanted messages.

Email itself is kinda broken this way. The costs of sending are that small that it profits people to send out millions of untargeted emails in the hope that they get a response from that fraction of a percent that might bother to read it.

There have been a number of attempts to redress this balance but none of them have caught on outside of a tiny audience.

Personally I gave up trying to fight it head on. I use a lot of spam filtering and manual white and blacklisting which makes email usable for me, but for everyday use I use instant messaging to communicate with collegues, email only being used for big messages or forwards from other people – and I like it this way.

If something is urgent, then email is useless, the feedback loop is too open. It needs to be something more immediate in response. IM works here and for when it doesn’t work for one reason or another, I use Textsecure a secure SMS replacement app for android.

This isn’t perfect, but it is working for me so far.

Bi-phasic sleeping

Now that I don’t have to get up for work at a fixed time, my body seems to be heading towards a biphasic state; which I am told is natural for humans and what our ancestors did.

Typically this means that I go bed about 10 or 11pm and sleep for a few hours; typically 1.5-3 hours correlating to 1-2 REM cycles. Get up and work for a few hours and then head back bed for another REM cycle or sometimes 2 if I am really tired. This means I am naturally getting 4.5-6 hours sleep a night and not feeling particularly tired at the end of it.

I have to admit, this is a nice state to be in some respects, but it isn’t helpful in others as I am out of sync with my girlfriend who is on a fixed schedule with regards to uni.

Moving on

After tomorrow I wont be an employee of Synety anymore.

In some respects this is a good thing; I wasn’t in a good place head-wise, but I will miss the place.

I am thankful for my time there, I have learnt a lot about myself and faced many challenges I hadn’t faced before in my career.

I might not have agreed with every decision that was made, who does, but I liked the company. I liked the people I worked with, and I enjoyed most of the work. I just needed a change; a break from the stress.

Synety itself is fast paced. It has needed to be to grow as fast as they have, but that speed of change takes it toll. While I could have coped with it had I not had other stresses in my life at the time, for me it was just too much.

You don’t realise it at the time, but stress changes you. It upsets your health. It upsets your relationships with friends and family and left unchecked it will change you as a person.

I wish them well in their future business dealings, but for me it is on to pastures new.




I recently implemented gravatar on a site for a friend and he was worried about the security of the 3rd party service.

I told him at the time the biggest issues are likely to be confirmation attacks and leakage attacks.

Fast forward to today and it I happen across this post.

What it comes down to is that a lot of the display names people use on a site are similar to the user part they use on their email addresses. Combine this with the ability to check the top providers (gmail/yahoo/hotmail etc) easily and you can typically confirm that an email address is genuine or not using offline methods by comparing the hash in the page to the hash from the calculated hashes for the username and provider.

I’m not sure that there is an easy solution here.

You could ask the user to add a new email address for the site and associate the picture with it. You’d only leak the site email then, but this is extra overhead for the user.

You could ask the user to upload their own image to the site and opt-out of gravatar, but both of these negate one of the reasons of using it in the first place – to reduce user effort.

Much that I dislike the data-creep of big corps like facebook, google or twitter, I think they all have it right in that you use a site specific id that is unrelated to your email. This mostly prevents external leakage, but you are still leaking a lot of information to the identity provider.