Sergey Shinder - DEV Community

Sergey Shinder

Sep 20

Our rate limit punished everyone except the customer causing the problem

#sergeyshinder #sre #reliability #ratelimiting

2 min read

Sergey Shinder

Sep 20

Nine teams were taking turns on one staging environment

#sergeyshinder #releaseengineering #devops #automation

1

2 min read

Sergey Shinder

Sep 20

We never told the partner which version of their API we wanted

#sergeyshinder #api #integration #versioning

2 min read

Sergey Shinder

Sep 20

We spent the first twenty three minutes of an outage reading staging's graphs

#sergeyshinder #observability #grafana #monitoring

2 min read

Sergey Shinder

Sep 20

Our cluster had kept every job it ever finished

#sergeyshinder #kubernetes #containers #devops

1

2 min read

Sergey Shinder

Sep 20

An empty list in our module meant every bucket

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 20

Every merge in the company depended on the dev cluster staying up

#sergeyshinder #cicd #testing #devops

1

2 min read

Sergey Shinder

Sep 19

Nobody had timed the part that comes after the restore

#sergeyshinder #sre #reliability #backups

2 min read

Sergey Shinder

Sep 19

Five percent of our traffic was not five percent of our callers

#sergeyshinder #releaseengineering #deployment #devops

2 min read

Sergey Shinder

Sep 19

Our totals were wrong by pennies and the pennies were in the parser

#sergeyshinder #api #integration #json

2 min read

Sergey Shinder

Sep 19

One new label multiplied our metrics by every order we take

#sergeyshinder #observability #prometheus #monitoring

2 min read

Sergey Shinder

Sep 19

The configuration we changed in June arrived one pod at a time in August

#sergeyshinder #kubernetes #containers #configuration

2 min read

Sergey Shinder

Sep 19

Terraform adopted the database and set it back to our defaults

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 19

Our pipeline tested the files we touched and none of the code we broke

#sergeyshinder #cicd #monorepo #devops

2 min read

Sergey Shinder

Sep 18

The queue had no limit so our timeouts meant nothing

#sergeyshinder #sre #reliability #architecture

2 min read

Sergey Shinder

Sep 18

The migration finished in a second and the rollout ran for eleven minutes

#sergeyshinder #releaseengineering #database #devops

2 min read

Sergey Shinder

Sep 18

Their certificate worked in every browser and failed in our service

#sergeyshinder #api #integration #tls

2 min read

Sergey Shinder

Sep 18

Averaging our percentiles gave us a number nobody had experienced

#sergeyshinder #observability #monitoring #metrics

2 min read

Sergey Shinder

Sep 18

Our pods slept for sixty milliseconds out of every hundred

#sergeyshinder #kubernetes #containers #performance

2 min read

Sergey Shinder

Sep 18

Deleting one subnet from a list rebuilt six of them

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 18

The same commit built us two different images

#sergeyshinder #cicd #docker #devops

2 min read

Sergey Shinder

Sep 17

We announced read only mode and could not even sign in

#sergeyshinder #sre #reliability #database

2 min read

Sergey Shinder

Sep 17

A one line fix took three weeks of unreleased work to production

#sergeyshinder #releaseengineering #git #devops

2 min read

Sergey Shinder

Sep 17

On the second of January we started overwriting last year's orders

#sergeyshinder #api #integration #database

2 min read

Sergey Shinder

Sep 17

The timeline we built from the logs was physically impossible

#sergeyshinder #observability #logging #sre

2 min read

Sergey Shinder

Sep 17

Half of our new capacity could not run our image

#sergeyshinder #kubernetes #containers #devops

2 min read

Sergey Shinder

Sep 17

Every apply we ran was a small one on purpose

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 17

Two formatters disagreed and the branch built itself all night

#sergeyshinder #cicd #githubactions #automation

2 min read

Sergey Shinder

Sep 16

The admission webhook rejected every change, including the one that would fix it

#sergeyshinder #kubernetes #sre #reliability

2 min read

Sergey Shinder

Sep 16

Eight months in production and no automation had ever touched it

#sergeyshinder #releaseengineering #automation #devops

2 min read

Sergey Shinder

Sep 16

Our partner calls failed more often the quieter it got

#sergeyshinder #api #integration #http

2 min read

Sergey Shinder

Sep 16

Eighty five percent full gave us seventeen minutes

#sergeyshinder #monitoring #alerting #sre

2 min read

Sergey Shinder

Sep 16

The debug copy of our service was taking one request in three

#sergeyshinder #kubernetes #containers #devops

2 min read

Sergey Shinder

Sep 16

A firewall rule with two owners changed itself back every three hours

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 16

Our test runner had one core and the race needed two

#sergeyshinder #cicd #testing #concurrency

2 min read

Sergey Shinder

Sep 15

Every new replica took twenty more database connections

#sergeyshinder #sre #postgres #reliability

2 min read

Sergey Shinder

Sep 15

A third of our releases went out through the emergency path

#sergeyshinder #releaseengineering #devops #cicd

1

2 min read

Sergey Shinder

Sep 15

Their sandbox accepted everything their production refused

#sergeyshinder #api #integration #testing

2 min read

Sergey Shinder

Sep 15

Nobody could tell me when the latency had started

#sergeyshinder #observability #monitoring #performance

2 min read

Sergey Shinder

Sep 15

A temporary file on one pod evicted every other pod on the node

#sergeyshinder #kubernetes #containers #storage

2 min read

Sergey Shinder

Sep 15

The apply was green and nothing in the environment could get out

#sergeyshinder #terraform #iac #networking

2 min read

Sergey Shinder

Sep 15

The secret was masked until we base64 encoded it

#sergeyshinder #cicd #security #githubactions

2 min read

Sergey Shinder

Sep 13

One instance was bad at its job and nothing was designed to notice

#sergeyshinder #sre #reliability #loadbalancing

2 min read

Sergey Shinder

Sep 13

Our own retention policy deleted the version we needed to roll back to

#sergeyshinder #releaseengineering #docker #devops

1

2 min read

Sergey Shinder

Sep 13

We read the partner's file while they were still writing it

#sergeyshinder #integration #api #sftp

2 min read

Sergey Shinder

Sep 13

Our dashboards were green because the request never reached us

#sergeyshinder #monitoring #observability #frontend

2 min read

Sergey Shinder

Sep 13

The sidecar left before the application had finished

#sergeyshinder #kubernetes #istio #containers

1

2 min read

Sergey Shinder

Sep 13

An environment was two settings that had to agree

#sergeyshinder #terraform #iac #devops

1

2 min read

Sergey Shinder

Sep 13

Each pull request was tested against a main that no longer existed

#sergeyshinder #cicd #git #testing

1

2 min read

Sergey Shinder

Sep 12

Nothing could start without the dependency we had filed as optional

#sergeyshinder #sre #reliability #architecture

1

2 min read

Sergey Shinder

Sep 12

The version number that promised nothing had changed

#sergeyshinder #versioning #semver #automation

1

2 min read

Sergey Shinder

Sep 12

A new status value from our partner rejected the whole batch

#sergeyshinder #api #integration #json

1

2 min read

Sergey Shinder

Sep 12

Somebody silenced that alert in March and it stayed silenced

#sergeyshinder #monitoring #alerting #oncall

2 min read

Sergey Shinder

Sep 12

Our workers were idle and the queue was an hour deep

#sergeyshinder #kubernetes #autoscaling #performance

1

2 min read

Sergey Shinder

Sep 12

We told Terraform to ignore one field and it hid a firewall change

#sergeyshinder #terraform #iac #security

2 min read

Sergey Shinder

Sep 12

Two deploys raced and the older commit won

#sergeyshinder #cicd #githubactions #deployment

1

2 min read

Sergey Shinder

Sep 11

The cold start we had never practised

#sergeyshinder #sre #reliability #incidentresponse

2 min read

Sergey Shinder

Sep 11

The deploy that finished while a third of the fleet stayed behind

#sergeyshinder #deployment #automation #aws

1

2 min read

Sergey Shinder

Sep 11

The 200 OK that meant the payment had failed

#sergeyshinder #api #integration #payments

1

3

2 min read

Sergey Shinder

Sep 11

The alert that never saw a spike shorter than its own window

#sergeyshinder #prometheus #monitoring #alerting

2

2 min read

Writing Debut