I am John and I solved a problem

I am John, I work for a company whose name I can not say and we have this client, but I am not allowed to mention their name, and we had this problem for which I cannot give specifics but I am here to tell you about how we implemented the solution.

Sounds familiar? Maybe, maybe not.

Well, for me this is the thing I hear at almost every conference I attend. Truth be told, I can understand that one can not give specifics, but one should also not start a topic meant to teach others unless you are willing to go into specifics.

I have the feeling that these “Jonhs” want to create the impression that they encountered a million problems and found at least one solution for each.

But how do I know that the problem was not created due to a faulty logic? How do we (the attendees) know if maybe the requirement was stated or understood in the wrong way. Usually that is where the problem stands. And it is best to ask all the questions before starting to work.

Anyhow, there are speakers for whom I would travel half the planet, but there are also others that have fancy job titles and do not have a clue of what’s going on in the dev world.

So, conference organizers: please, please I would rather see the same 3 speakers in all the conferences than having different poorly prepared speakers, just for the sake of not repeating.

And for the speakers: Please speak for the sake of sharing and making the community better not for the money. Remember why you became a programmer! If money is the first answer, you are doing it wrong.

I am John and I solved a problem

The consistent Petr (part 2)

In our project we have this Sentry integration that would let us know when something goes sideways (I am sure most devs know what I am talking about).

About a moth ago (all of devs and dev-ops) have missed one notification (we were all in a meeting an ignored the email) and live site was unusable.

Ever since I made a rule (for myself) to put the errors in a common Slack channel for all the devs to see. All good, no one was bugged.

A few days ago, Petr makes this announcement:

“Note – it seems that current errors from production are maybe false-positive and they are not affecting user. Sentry integration will be updated shortly.
Second note – even these errors may be false, we will still take a look into them.”

Today I see tow dozen errors in my email  and I notify the devs, that maybe something went wrong, as there are too many different ones. If they are nothing than only 5 mins of our lives were lost.

To my notification I get this from Petr:

Yes, the FE errors on production were mentioned on stand-up and also in #frontend channel. They can be false positive, due to complications with Sentry integration.

There was no mention of said errors in any of the stand-ups this week or past week.

The consistent Petr (part 2)

The consistent Petr (part 1)

This morning, when I got into the office, I was surprised by my tester who said that none of the bugs I’ve put in for testing are fixed. Imagine my amazement when I checked the bug-branch I was working on and the testing-branch and they had very different code! So I go to Petr and:

Me: hey Petr, did you get any conflicts when merging the bug-branch into the testing-branch
Petr: Yes. But I fixed them. I kept the testing-branch code
Me: You should have kept the bug-branch code as it was the newest one (and I added it to fix some bugs reported on the testing-branch)
Petr: Yes, yes. That is the one I used.

Half of the day has passed since and we still have different versions of code even though the bug-branch was merged into the testing-branch…Petr is on it (fixing).

The consistent Petr (part 1)

What is a senior developer?

There was a very interesting discussion in the office today about what a senior developer is (we were talking about JavaScript developers, but I guess it could be applied to any programming language).

I was saying that there is no possible way that a developer can be senior after only 2 years of experience. There is simply not enough time to meet the unexpected. And that is what, in my opinion, makes a senior: the unexpected. The ability to not be surprised and blocked. That is when a developer becomes senior.

There were of course voices that said that 2 years is more than enough. And it only depends on how passionate you are (how can passion be a factor in establishing seniority?). Then there was a slip from someone who said: if you convince your current employer to make you a senior, than you can put it in your CV and the next job recruiter would have to take you into account for a senior role. That made me think: hm.. sneaky one. So you you put it in your CV just for to market yourself? So the seniority became just a keyword? Who is to blame for this? Can we fix it?

About 3 years ago someone estimated that the number of programmers doubles every 5 years. I suppose then, there is this need of making oneself appear “seniorer” than the rest. But is it fair? and I mean it. There are senior developers with less than 2 years of experience and less than 25 years of life who participate in technical reviews during interviews (and I mean it because I have seen it) and they evaluate people that have more years of experience then they have of life, only to conclude that the respective developer is not good enough. I guess I am a bit afraid that I will one day be evaluated by one of these. And maybe I am more afraid of my reaction to it. I am afraid that I will try to impress this kid into hiring me. Is this ok? Is there something wrong with me? Are there any others like me?

 

What is a senior developer?

Interview questions

I was recently invited to an interview for a frontend developer role. After the interviewer asked me a lot of questions about http, http2, devOps, linking backend with frontend he said: now let’s go to the more unpleasant questions. And he started with: what is this small tag at the beginning of an HTML page <doctype... And he went on about DTD then we moved to local storage, then CSS and pre-processors …

When and why did HTML and CSS became unpleasant? how else would you develop for the web if not using HTML and CSS?

I am now asking you: people of the world that you call yourselves web developers, who are you and what did you do with the web developers?

Interview questions

Web developer definition

This article exists because a tweet is to short and @ppk does not allow comments in his articles (i.e. this article: “What is a web developer?“). And maybe is a good thing that he does not allow comments – this gave me the opportunity to write down my thoughts without using other people’s space.

For about a year I moved from web development to application development (big mistake, will explain in a different post). And the same “developer problem” is present here too.

I believe this is because there are more and more people that call themselves developers [or even worse: web architects] and they don’t know the first  thing about how the web works; they have no idea about what web standards are and why they exist.

Sadly most of them started their developer life – brace yourselves –  programming in jQuery, or React or Elm (that’s the latest..). And I ask you now: how can one “program” in jQuery? When did jQuery become a programming language?

I tried saying these things out loud, but I got hit very hard by the young ones.

Maybe part of the problem is the wrong/too young [please read not enough life/work experienced] people in the high management or in the hiring process on the technical level (I know I saw this one too many times in a big corporation I used to work for).

What happened to people fighting for a better web? Are we just saying it to be trendy? What’s so trendy about not knowing the basics about doing your job? Why did you chose to be a web developer if you do not know what www is?

PS: I am not as old as the article sounds, but I am very sad about what I see around me. And maybe one article is not enough to change something, but I see it as a start. Maybe it will push some button on someone.

Web developer definition

The king is dead, long live the king!

I read this article today: “Saying Goodbye to Firebug” and I remembered how me and Firebug first met. I thought it to be a good idea to write a few lines about it because Firebug was a good friend to say the least.

I remember the first time I saw Firebug. I was 2006, sometime at the beginning of the year (probably Jan or Feb). I was in the second year at the Faculty and it was the first session of a course called “Web techniques”. The first condition to pass the exam was to have Firefox and Firebug installed. No Firebug, no passing! You can not imagine what we all felt. How can a plug-in possibly help us! Little did we know…
And now, 11 years later, I a front end developer still using Firebug. To pass a different exam: the users acceptance.

I would like to thank you Firebug for all the help and support. You were there all the time for all the steps.
For tables and divs, for images and rounded corners, for transparency and animations. You helped a lot of people and brought up a bunch of great developers.

So thank you, friend.

The king is dead, long live the king!

Optimistic locking on MongoDB

There is a problem on the persistence side of our models: concurrent updates. Some of us apply the technique of last-one-wins, by not protecting against simultaneous modifications of an entity. This can be OK in most cases (like CRUD) but when we have calculations to do based on the previous values we encounter problems. I encountered this situation on my event-based architectures when the read models are updated as the events arrive.

But let’s start with an example: Let’s suppose that we have a simple product entity that can be rated and we only need to provide the average rating and the number or ratings. So, our model would look something like this:

class Product
{
    /**
     * @var AverageRating
     */
    private $averageRating;

    /**
     * @var ProductId
     */
    private $productId;

    public function rate(int $rating)
    {
        $this->averageRating = $this->averageRating->addRating($rating);
    }
}

class AverageRating
{
    private $value = 0;
    private $count = 0;

    public function addRating(int $rating): self
    {
        $other = clone $this;
        $other->value = ($rating + $this->count * $this->value) / (1 + $this->count);
        $other->count++;
        return $other;
    }

    public function getValue(): int
    {
        return $this->value;
    }

    public function getCount(): int
    {
        return $this->count;
    }
}

But if we do like this:

   public function onProductWasRated(ProductWasRated $event)
    {
        $product = $this->loadProduct($event->getProductId());

        $product->rate($event->getRating());
        
        $this->saveProduct($product);
    }

We would be in big trouble if two events arrive at the same time: both the product loading are done (lets suppose the previous rating count would be 100), then both add a rating (the new rating count becomes 101, for each operation) then the new rating is persisted (so 101 rating count is persisted, by the last operation); this is bad because there were 100 ratings and after two more arrived only one had effect. Please notice that the protection that the underlying database is providing us is not helpful (row locking, document locking, whatever).

So, how to add both of the ratings, even if they arrive at the same time? The answer is: by retrying the whole load-calculate-persist process. This is called optimistic locking.

But for this to work you have to detect concurent updates and not succeed if a previous update was done. For this you need to have a version property on your entity. This version is the version that existed when the entity was loaded and when you try to persist the changes to the entity you check that the current version is equal to that loaded version. If the stored entity has a new version then that means that you based your calculations upon an old version of that entity and you must retry.

In order to extract this algorithm into a class we make a Versionable interface like this:

interface Versionable
{
    public function getVersion();
}

Then, our product must implement this interface:

class Product implements Versionable
{
    /**
     * @var AverageRating
     */
    private $averageRating;

    /**
     * @var ProductId
     */
    private $productId;

    /**
     * @var int
     */
    private $version = 0;

    public function rate(int $rating)
    {
        $this->averageRating = $this->averageRating->addRating($rating);
    }
    
    public function getVersion():int
    {
        return $this->version;
    }
}

Our refactored product list would look like this:

    
    public function onProductWasRated(ProductWasRated $event)
    {
        $this->update(
            $event->getProductId(),
            null,//the factory, do not create the product if it does not exist before rating
            function (Product $product) use ($event) { //the updater
                return $product->rate($event->getRating());
            }
        );
    }

    private function update(ProductId $id, callable $factory = null, callable $updater)
    {
        $this->updater->addOrUpdate(
            $this->getCollection(),
            $id,
            function ($id) {
                return $this->loadProduct($id);
            },
            $factory,
            $updater,
            function ($entity) {
                return $this->serializer->convert($entity);
            }
        );
    }

It is OK to use not-type-hinted callables because this is a private manner. To use the same abstraction level in our event handler I have extracted a new method: private function update. This method accepts the ProductId, the factory callable (is called if the product does not exist yet) and the updater function that would perform the actual update on the product.

And now the updater:

class OptimisticMongoDocumentUpdater
{
    public function addOrUpdate(Collection $collection, $id, callable $loader, callable $factory = null, callable $updater, callable $serializer)
    {
        /**
         * We try to add/update the entity in a concurrent safe manner
         * using optimistic locking: we always try to update the existing version;
         * if another concurrent write has finished before us in the mean time
         * then retry the *whole* updating process
         */
        do {
            /** @var Versionable $entity */
            $entity = $loader($id);
            if (!$entity) {
                if (!$factory) {
                    return;//do not create if factory does not exist
                }
                $entity = $factory();
                $version = 0;
            } else {
                $version = $entity->getVersion();
                $entity = $updater($entity);
            }

            $serialized = $serializer($entity);

            unset($serialized['version']);
            $serialized['lastModified'] = new UTCDateTime();

            $result = $collection->updateOne(
                [
                    '_id'     => new ObjectID($id),
                    'version' => $version,
                ],
                [
                    '$set' => $serialized,
                    '$inc' => ['version' => 1],
                ],
                [
                    'upsert' => true,
                ]
            );
        } while (0 == $result->getMatchedCount() && 0 == $result->getUpsertedCount());//no side effect? then concurrent update -> retry
    }
}

Pretty nice, right? 🙂

So, the idea of this updater is that it tries to update the know version of an entity. If that version does not exist anymore, then the whole process is restarted.
Another important detail is that the version is incremented atomically, in the same time as the actual update, using the $inc operator.
You have now a concurrent-proof MongoDB entity updater 🙂

Optimistic locking on MongoDB