docker_logdna—deep dive into Docker plugins
Christopher Besch (IBM) • 9th February 2024
docker_logdna
Deep Dive into Docker Plugins
What we'll talk about
- What is the docker_logdna Plugin?
- Performance
- Testing
- Logspout Alternative
What is a Log Aggregator?
- Example:
- Authentication request arrives at proxy -> proxy logs original IP
- LDAP only sees proxy's IP -> LDAP logs rejection of request
- Proxy and LDAP logs need to be combined
- Solution: ELK, Graylog, Fluentd, Logdna, ...
Why Logdna?
- IBM Log Analysis uses it
- We don't want to host own aggregator
What is a Docker Logging Plugin?
docker run --log-driver fluentd debian
Overview
What do I have to do
- Check installed docker_logdna version with
- Add logging section to your docker-compose.yaml
- Documentation on github.com/IBM/docker_logdna
$ docker plugin ls
Performance
©NASA 1969
- Never block the application!
- Read logs from fifo file ASAP!
- Low resource consumption
- No resource consumption when blocked waiting
Simple Architecture
Problems
- Multiple Container can't be handled
- Busy waiting wastes CPU time
Solution: Async
- Run tasks concurrently
- Not parallel with multithreading
- Great for IO-bound tasks
Introducing the Future
- Code to be executed when the time is right
- Can await external condition
- Async runtime schedules polling of futures
Simple Architecture
Async Architecture
Testing
Let's get that Green Color!
What do we Want?
- Handle edge cases
- Handle rare conditions (stress, external environment slow, ...)
- Prevent regressions
- Consistent results
- Errors easy to debug
How do we get there?
- Unit tests (white box)
- End-to-End tests (black box)
- Automated at every change
- Fast (short turnaround time)
- Full control over environment (i.e. tests not dependant on external API)
Is docker_logdna easy to test?
The Good
- Little state
- Almost entirely linear pipeline
- No UI/UX
The Bad
- IO-heavy code
- External input/output difficult to mock
What to test for
Unit Tests
- Config parsing
- Docker HTTP socket interaction
- Application Container protobuf fifo file interaction
- Logdna interaction
End-to-End Tests
- Many log lines per second
- Long log lines
- Many containers
- Unresponsive Logdna server
Mocking
- Minimize differences between mock and original
Unit Tests
Unit Tests
- Test all edge cases
- At least one unit test per outcome
End-to-End Tests
- Use Docker-in-Docker
- full control over environment
- don't clutter host with 100+ containers
End-to-End Tests
Overview
- ~900 lines of code
- ~1600 lines of tests
Comparison to Logspout
docker_logdna Plugin
- Docker plugin (one per Docker install)
- Rust
- max. 60.0k lines/sec
- github.com/IBM/docker_logdna
Logdna Logspout
- Container with Docker socket access (one per ingest server)
- Golang
- max. 41.8k lines/sec
- github.com/logdna/logspout
Coming Soon: IBM Open-Source Approval Talk
Coming Soon: IBM Open-Source Approval Talk