Early in 2015 I was given the opportunity to work in Betfair’s BLIP site in Porto. I’ve been around both Betfair and software development for some time, more about me here, and it was felt that experience would be useful around the place.
I have a young family and a wife with a challenging job but thankfully with long holidays so the opportunity wasn’t impossible to take but it wasn’t easy. What sold it to me was Blip. Blip, Blip, Blip, what can I say about the place that hasn’t already been said. Google “Blip.pt” and hopefully you’ll get a feel for the place. I’d previously handed over a project from Betfair’s development base in Hammersmith to Blip and knew the level of energy and enthusiasm that was waiting for me.
The logistics of the trip were difficult, planning around family life and sticking within budget are never easy but with some help from “Star Trek” – the team who sort out all the office management / HR stuff in Blip – and especially from Pedro T we found a flat by the beach about 20km North of Porto which I could cycle to the office from and where the family would be happy. See for yourselves ….
Working life was both fascinating as well as hugely challenging. My role at Betfair can only be described as middle management. I have a long track record as a PL/SQL developer and wrote swathes of what was for many years Betfair’s intellectual property. But that is no more, the Bet Matching is no longer in the database and the exposure calculation will be out anytime now. However I’ve never managed a team where I didn’t understand the underlying data structures and couldn’t roll up my sleeves and help out if deadlines were looking tight. All of a sudden I’m asked to work with 3 teams, our SRE team, the legendary #! team, who are NOT DevOps , and the Man O’War team building a new feeds platform on Kafka streaming with Storm topologies. My archaic tech skills were going to be of no use here, SRE and #! are supporting nearly 200 web and mobile developers.
The first few weeks are very hard, I’ll be honest, my head hurts every day after work. Everyone is extremely friendly and helpful but I don’t know the tech, I don’t understand the relationships in the office, I don’t understand 90% of what’s said around me and, despite the Portuguese lessons that Blip laid on for me, I can’t get a grip of the language. The language really frustrates me, I’m French and when I worked in Italy I picked it up to a conversational level really quickly. I can read a CV in Portuguese with relative ease, but I can’t understand a bloody word. On top of all that I have 2 bosses, that’s always an interesting balancing act.
Does it get better? Yes. Two things happen, I realise that I’m going to need to be useful to the teams, I have my objectives from higher up but those aren’t going to get done unless the teams see me as an asset. I throw away my copy of “The first 90 Days” and started looking for gaps to plug.
With the Man O’War guys we get a lucky break, the project to rewrite our generic feeds ingestion platform had been treading water for some time and it wasn’t clear why we couldn’t break the back of it. It is here that Betfair acquire a current Tennis feed for Wimbledon, one of Betfair’s biggest events. Can we get our Tennis feed up and running, fully automating our entire Exchange Tennis offering and feeding information to our bot writers in time for the Qualifying rounds? Here is something I can help with, I have worked on the original system that was being re-written, I know the other teams that we’ll need to work with in Hammersmith, I know the product people and I know the Exchange Operations team in Stevenage. We get together as a team and I ask them if they think they can achieve this in time …. “No” comes the answer. What to do? Well it strikes me that if we align all our efforts for Wimbledon but fail we would still have broken the back of the problem. I rephrase the question – well I ask a different question – Can we aim to achieve this in time, “Yes” comes the answer. And they make it too. Three of us worked through the São João celebrations which are huge in Porto (see below) and at times it was a bit scary but they pulled it off. I will always remember that team and what they achieved fondly as we fostered a real spirit of camaraderie over that period.
And what of the SRE and #! (NOT DevOps) teams. Here my objectives from above are different, neither team is seen as stuck, in fact quite the opposite, both are highly regarded by the development teams that they support. I’m received really well, super helpful and very friendly but it strikes me quite soon on that no one is asking anything of me, things would tick over just fine without me being there it seems.
However over time I am able to get involved in the life of the teams. One of the first points is that Betfair and Blip were / have moved to what becomes known as a “Devs On Call” pattern. The theory (and it’s one I whole heartedly support) is that each dev team should support their own component in production, 24/7 and have all the training and access needed to do so. To support this a budget had been set aside for on call rotas and each team signed up to PagerDuty for call outs. However anecdotally something isn’t right, the #! guys seem to be being called a lot still, what is going on? I pull some numbers out of the HR system (management does have its privileges!) and am surprised that only two teams have been called in the 6 weeks up until then. The process between the teams and the call outs is broken. What to do? Well we steal a solution from Netflix, and essentially create our own Chaos Monkey and start creating managed, non-customer impacting, degradations to see how the teams respond. This highlights a number of issues, from incorrectly configured alerts to a lack of connectivity to Pager Duty and team members not having the prerequisite access or knowledge. For those of you who saw João Pinto’s presentation at DOCCON 2015 the rest, as they say, is history.
All in all, it was a memorable experience both professionally and personally. I came back from time @Blip with a broader outlook, an understanding that there are yet more ways of working efficiently than I’d worked with before . A vision of how an Ops team can be almost symbiotically related to the development teams that they support and a new found interest in all things Ops. And a, very, few words of Portuguese.
Within many of the dev teams at Betfair we run a mini Erasmus programme where we swap team members between Blip and the London office, if you’re in one of those teams and have access to this I suggest you snap up the opportunity ASAP. You will discover a different way of working, meet stakeholders and colleagues and bring back new ideas, working practices and leave a little of your own fairy dust behind.
If you don’t work for Betfair but do work in an organisation with multiple sites, do you do this or something similar? Would a placement or an exchange be something you can sponsor or request? What are your experiences with this, I’d be keen to hear from others with similar stories to tell. Finally if your current work can’t or won’t give you opportunities like this, well we’re always hiring good people, come and check us out here: http://www.betfair.jobs/ and here http://www.blip.pt/
/Thomas Andrew is a Global Senior Delivery Manager in our IAE Team