Categories
Uncategorized

2021W36

Most of this week was follow-on from last week’s attempt to make a data pipeline read from a Spanner database. The configuration turned out to be quite the tricky thing to make work. But at least I solved it for everyone. And the improvements are worthwhile: the pipeline runtime has halved, and we’re now able to read much more recent data, as we’re not relying on exports of database backups. The key part to making this work is that the pipeline is using a new feature to access the underlying files that the database uses, instead of having to make an online query.

Aside from that, one highlight was a pair programming exercise with a colleague in the USA who was interested in learning Go. We managed to contribute an improvement in about an hour, which is great, and he’s now looking at making changes on his own.

I spent a great deal of time with my manager writing up a summary of what I’d done over the last six months. I find it hard to present a coherent, data driven picture of this (despite the effort of writing snippets each week, so I actually have something to base this on…)

Stats: 18 changes submitted, 47 reviewed.

Categories
Uncategorized

2021W35

This week was fairly quiet: Monday was a bank holiday, and it’s perf season inside Google, so everyone was busy writing about themselves and others instead of writing code…

Unfortunately for me personally it was a week of going down rabbit holes & yak shaving.

I spent far, far too long visiting the innards of cmp and protocmp in an effort to modernise an older test. The premise was simple: convert all int64 values in a particular message to 1, so that we don’t care about the values, just the presence or absence of the value. I was trying to use FilterMessage(), but couldn’t make it work. Eventually I ended up with a more brute-force approach: alter the entire result, not just the message I was concerned with.

cmp.Transform("normalize", func(int64) int64 {
  return 1
})

The other yak shaving relates to a data pipeline. I was switching it from reading some files to reading from a Spanner database. The code change was verbose, but fairly simple. Then I tried to run it and discovered the configuration change I needed was anything but. I ended up with some configuration of command line flags … in a proto message … wrapped in another command flag … embedded in some GCL … in turn embedded in another shell script. Not only was the quoting horrific, but I couldn’t make the GCL do what I needed. I ended up hacking the interpreter to emit errors from an eval() statement which are normally thrown away. This is one of the benefits of having everything in a monorepo, but I really didn’t need to spend hours on this.

Stats: 9 changes submitted, 47 reviewed.