Monday, December 16, 2019

Notes on Agile Retrospectives

I recently read (well, skimmed through) the book Agile Retrospectives - Making Good Teams Great by Esther Derby and Diana Larsen. The authors share lots of insight & experience on having retrospectives.

What is a retrospective? Quote from the book:

When we say retrospective, here's what we have in mind: a special meeting where the team gathers after completing an increment of work to inspect and adapt their methods and teamwork.

Some benefits of retrospectives the authors have heard:

  • Improved productivity
  • Improved capability
  • Improved quality
  • Increased capacity

Typical structure for a retrospective

  • (1) Set the stage
    • Welcomes & appreciation of people's time
    • Duration, goal & purpose of the retrospective
    • If the team has working agreementa, post and review them.
  • (2) Gather data
    • Ask "What?"
  • (3) Generate insights
    • Ask "Why?"
    • Think what to do differently.
  • (4) Decide what to do
    • Plan experiments and actions
    • Pick only the top items (for an iteration retro, 1-2 could be enough)
    • Check out that people sign up and commit to the selected tasks.
  • (5) Close the retro
    • Decide how to document what the team has learnt
    • Plan for follow-up

Tailoring a retrospective for your team - Planning the retro

Knowing the context

When preparing the retro, aim to have answers to the questions:

  • What is the context of the team?
    • If you're working with your own team, you probably already know the history & context of your team-
    • If you're working with a team other than your own, study the context. (To get clues about what questions to ask and what challenges the team might have.)
  • What's the goal for the retro?
  • How long will the retro be?
  • Where will the retro be hold?
    • A setup where everybody can see other people's faces is preferred.
    • Whiteboards for post-its, walls for timelines, flip charts etc.
  • What's the structure of the retro?
    • How much time for the different phases?
    • If the retro is longer than two hours, remember to have break(s)

Selecting activities for the phases

Based on that you can select the activities for different phases.

  • The book has a good selection of activities for each phase.
  • The book author noted that ARCS criteria for evaluating instructional designs can be applied also for retro activities:
    • Attention
    • Relevance
    • Confidence/Competence (activities that the people can complete successfully)
    • Satisfaction
  • As a tip, you can choose alternative activities (longer & shorter) to help with managing the time.

Leading a retro

  • Facilitator vs. participant
    • As a facilitator your primary responsibility is the process.
    • Participants focus on the content, discuss and make the decisions.
  • Managing the activities
    • As a tip, when using an activity for the first time, write a script for yourself.
    • If multiple parts, give the details for each part.
    • After giving instructions, ask for questions, pause and count to ten.
  • Two main tasks during the activity:
    • Answer questions about the activity
    • Monitor the room
  • Debriefing the activity, e.g. with the following:
    • Ask "What did you see and hear?"
    • Ask how people responded: "What surprised you? Where were you challenged?"
    • Ask for insight & analysis
    • Ask how could the insights be applied?
  • Related to group dynamics:
    • Aim for everybody to participate. Make sure
      • People with something to say have the chance.
      • People with lots to say don't dominate.
    • After participating, next most common issues are violating the working agreements & blaming.
  • If team become stuck, you can e.g. ask something like
    • What have we tried before? What happened? What would you like to happen differently?
    • If we had that, what would we gain?
    • Have you ever tried this a different way? What happened?
  • Managing time
    • Have some timepiece to time activities etc.
  • Managing you
    • If needed, remember to take a deep breath.
    • Take a break if needed
      • Shake out your hands & legs
      • Take three deep breaths to get oxygen to think clear

Activities for the different phases.

The authors have a good selection of activities for each phase. Please check out the book yourself.

In addition, here are some links to collections of retrospective activities etc.:

Wednesday, November 27, 2019

Notes on Lambdacast, Category Theory etc.

I recently finished listening through LambdaCast, "a podcast about functional programming for working developers". Some notes on statically typed functional programming, basic category theory etc. will follow. I concentrate here mainly on the topics I haven't been using so much -> thus skipping some of the topics, focusing mainly on the category theorish parts. The podcast described category theory as the "math of maths".

On notation

I use here mainly Elm/Haskell-ish notation.

Parametric types

More notes later, but I'll mainly use Elm/Haskell-ish notation for parametric types:

  • List a means a list of values of type a, e.g. List String or List Int.
  • In Java/C# these are written e.g. List<String> and in Scala as List[String]
  • More on this below under "Polymorphism & type parameters".

Function type signatures

  • f: A -> B -> C means a function f that takes two parameters of type A and B and returns a type C.
  • With partial application, this can be interpreted as a function that takes parameter of type A and returns function of type B -> C.
  • So f: A -> B -> C means the same as f: A -> (B -> C).

Some examples

  • List.length: List a -> Int - function that takes a list and returns it's length as integer.
  • List.filter: (a -> Bool) -> List a -> List a - function that takes a "predicate function" and list and returns a new filtered list

Note that Haskell has roughly same notation but with two colons, e.g. f :: A -> B -> C.

Applying function

Write the function name followed by parameters. Note that no parentheses are used (as in e.g. JavaScript/Java/C#). For example, if we have addition function add: Int -> Int -> Int, it would be applied as add 1 2, evaluating to 3.

Function composition

Function composition is mentioned a couple of times. I'll use Haskell's infix dot notation, described by an example:

Let's say we have the two functions:

string2int :: String -> Int
int2bool   :: Int -> Bool

We could chain the functions as (assuming we have a string str):

myBool = int2bool (string2int str)

This can be also written with function composition .:

myBool = (int2bool . string2int) str

Various terms

Domain and codomain

If there is a function fun: A -> B (that takes value of type A and returns a value of type B)

  • All possible input values (A) are called domain of the function
  • All possible result values (B) are called codomain of the function.

Note that domain here is different term than in domain-driven design.

Associative property

Associative property is a property of a binary operation. Roughly, for a binary operation it means that (x ⊗ y) ⊗ z = x ⊗ (y ⊗ z). This holds true for example for addition, e.g. (2 + 3) + 4 = 2 + (3 + 4).

Types, categories and sets

These are somewhat related concepts and not totally identical. I'm going to cut quite many corners and mainly use the term type.

Morphisms (Episode 8)

Some corners cut, a morphism is a mapping/relationship from a type to another. The word comes from ancient Greek morphe (“form, shape”). In programming context, a function f: A -> B from type A to type B is a morphism.

Note that the morphism can have same source and target type. Endomorphism (from ancient Greek endon, “inner, internal”) is such a morphism. Such a function would look like fun: A -> A, for example. As a practical example, function inc: Int -> Int (that increases it's argument by 1) is an endomorphism with type A being Int. For a more mathematical description, see Endomorphism in Wikipedia.

Isomorphism (from ancient Greek ísos, “equal”) is a morphism that has an "inverse morphism". This can be also described as "no data is lost in the transformation". With types A and B that would mean that there are functions fun1: A -> B and fun2: B -> A such that fun1 . fun2 = identity (i.e. fun1 (fun2 x) = x). For a more mathematical description, see Isomorphism in Wikipedia.

Homomorphism (from ancient Greek homós, “same”) is a morphism that "preserves structure", e.g. List.map or List.filter For a more mathematical description, see Homomorphism in Wikipedia.

Catamorphism (from ancient Greek katá, “downwards”) is a morphism from an algebra to another. List.fold was mentioned as an example. For a more mathematical description, see Catamorphism in Wikipedia. Dual/opposite of catamorphism would be Anamorphism (from ancient Greek aná, “on, up, above, throughout”). Cata- & anamorphisms (as well as pylo- & paramorphisms) are discussed within FP context in the paper Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire (link in the end)

With morphisms, denotational design/algebra was also mentioned. That would concentrating on the verbs and nouns of an "algebra" that would be e.g. HTTP requests. This sounded somewhat related to DDD. The following presentation was recommended: Denotational Design: From Meanings To Programs.

Polymorphism & type parameters (Episode 9 & Episode 21)

The name polymorphism comes from ancient Greek polús (“many"), indicating something that can have multiple different shapes.

First, in some contexts, especially object-oriented programming, polymorphism means usually subtype polymorphism. A common example of this is class hierarchy of classes Animal, Cat extends Animal and Dog extends Animal where instances of Dog and Cat are also instances of Animal. This is quite a different concept than described here.

In functional programming context (with static types), polymorphism means usually parametric polymorphism. This means having types that have type parameters. Java/C# generics is quite the same concept, some corners cut. Typical example would be list that is parameterized to contain elements of certain type.

In e.g. Elm/Haskell this is written as List a, in Scala List[A] and in Java/C# List<T>. As a detail, in Haskell the type parameter is typically written in lower case whereas in Scala/Java/C# it's typically written in upper case.

Semigroups and monoids (Episode 12)

A semigroup is type with a binary associative operation that combines values of the type, resulting to a value of the same type. This operation is often called append or concat. Function signature would be append: A -> A -> A.

Many semigroups have an "identity" element e such as e ⊕ x = x ⊕ e = x for every x in the semigroup. A semigroup with an identity element is called a monoid.

Some examples of monoids:

  • Integers under addition, 0 being the identity element
  • Natural integers under maximum operation, 0 being the identity element
  • Lists under append operation, empty list [] being the identity element
  • Strings under concat operation, empty string being the identity element

Algebraic data types (ADTs) (Episode 13)

Algebraic data type system is a type system, where types are derived from other types as product types & sum types.

Product types ("AND" types) are pretty much like typical struct/class types, a combination of values, e.g.

type Card = Card
    { suit : Suit
    , rank : Int
    }

The term product type comes from the idea, that all possible values of Card can be originated as multiplying all possible values of Suit with all possible values of Int.

Sum types ("OR" types) are types that have alternative "choices".

Following the playing card example, a very simple example would be Suit

type Suit = Diamond
          | Heart
          | Club
          | Spade

So far this is like enums in many programming languages. What makes sum types more powerful is the possibility to add data to the values. Our earlier Card example did not support jokers but let's redefine it with a sum type:

type CardV2 = StandardCard Suit Int
            | Joker

With this, a StandardCard value will consist of suit and rank, in addition to it's type. The term sum type comes from the idea, that all possible values of Card2 are possible values of StandardCard + Joker.

Functors (Episode 16)

Very informally, a functor is a "containerish structure" (polymorphic type f a) that has a map operation (in Haskell world fmap), with the signature map: (a -> b) -> f a -> f b.

A "mathematical" functor follows two basic rules:

  • Applying map with identity function keeps the functor as it is: map id f == f
  • Applying map is composable: map (fun1 . fun2) x == map fun1 (map fun2 x)

List & Maybe are typi<cal examples of functor.

It's good to note that in mathematics context, the term functor is used mainly for the function/mapping, whereas in FP context the term is used more to describe the type that has map/fmap operation.

Note that partial application for map with a function (a -> b) results in a function of type (f a -> f b) that works "inside the functor" - this is called lifting.

Applicative functors (Episode 17)

Applicative functor is an abstraction between functor and monad, being somewhat new addition in Haskell world.

For an applicative functor type, there are two main operations (in addition to functors map/fmap)

  • Function for wrapping a value inside applicative functor: pure: a -> f a
  • apply: f (a -> b) -> f a -> f b
    • Note that in Haskell apply is named <*>

apply might feel a bit strange at first but with partial application that gives a possibility to apply a binary function to values inside two applicative functors.

As a simple example, we have two Maybes containing ints and we want to add them up (resulting up to a Maybe)

  • We can map Maybe Int to Maybe (Int -> Int) (e.g. having Just 5 would result as a function of one argument that increases it's argument by 5)
  • The Maybe (Int -> Int) can be given to apply with another Maybe, resulting as Maybe with sum.

Monads (Episode 18)

Monad adds flatMap on top of applicative functor. Type signature is flatMap: m a -> (a -> m b) -> m b. In Haskell context, flatMap is named >>= and often called "bind".

  • This kind of "forces" the operations to be sequential
  • Haskell has "do notation" that makes it more convenient to work with monads.

Monads are not so complex topic to get grasp as they might sound. They have many applications

  • Has lots of use with lists & maybes/optionals
  • Used often with futures and promises so that the result from previous async operation is used to initiate the next one
    • E.g. first do a HTTP request to fetch users and after that another to get some extra resources for the users.
  • Haskell as IO monad stating explicitly that something has side effects / state

Monday, November 11, 2019

Notes on Two Systems - first part of Thinking, Fast and Slow

Notes on first part of Thinking, Fast and Slow (Two Systems)

.

The characters of the story

  • Two modes of thinking: System 1 (Fast thinking / automatic system) and System 2 (Slow thinking / effortful system). These terms were originally proposed by psychologists Stanovich & West
    • System 1: Operates automatically and fast, with no or little effort, no sense of voluntary control
    • System 2: Conscious, explicit, effortful thinking, concentration.
  • System 2 has some ability to change the way System 1 works.
  • System 2 can influence where the attention is allocated (see e.g. "The invisible Gorilla")
  • System 1 works automatically, System 2 is usually in a comfortable low-effort mode
  • System 1 can "call System 2" when e.g. a question arises for which System 1 doesn't offer an answer
  • Illusions (both visual and cognitive)
    • e.g. Müller-Lyer illusion
    • System 2 can choose to believe that the lines are of equal length but it can't (at least without lots of training) prevent System 1 from seeing the lines to be of different length.
  • NOTE: System 1 and System 2 are fictious characters

Attention and effort

  • System 2's operations are effortful.
  • as a not: Size of the pupils indicate mental effort
  • If System 2 is under load, attention to other tasks (also of System 1) is reduced.
    • "... people, when engaged in a mental sprint, may become effectively blind" (e.g. "The invisible Gorilla")
  • Switching from one task to another is effortful.
  • Time pressure is another driver of effort

The lazy controller

  • For walking, there is usually "a natural speed" with which no strain or "need to push" is experienced
    • Also System 2 has a "natural speed"
  • System 2 follows "the law of least effort"
  • Maintaining a coherent train of thoughts requires discipline
  • Note: "flow" (studied by Mihaly Csikszentmihalyi)
    • People describe flow as "a state of effortless concentration so deep that they lose their sense of time, of themselves, of their problems"
  • A well-established proposition: Both self-control and cognitive work are forms of mental work (by System 2)
    • They both draw at least partly on a shared pool of mental energy (studied by Roy Baumeister)
  • Ego depletion
    • Effort of will or self-control is tiring.
    • If you have had to force yourself to do something, you are less willing or less able to exert self-control when the next challenge comes.
    • Study: Tired and/or hungry judges tend to fall back on easier solutions.
  • One of the main functions of System 2 is to monitor and control thoughts and actions "suggested" by System 1.
  • Recurrent theme of the book: Many people are overconfident, prone to place too much faith on their intuitions. (cognitive effort with System 2 is at least mildly unpleasant)
  • Stanovich & West have studied: What makes some people more susceptible than others to biases of judgement?
    • Stanovich draws a sharp distinction between two parts of System 2:
      • "algorithmic mind" (slow thinking and demanding computation) and "rational mind" ("engaged", more skeptical to their intuitions)
      • rationality should be distinguished from intelligence

The associative machine

  • System 1 does associations automatically - thoughts, memories, even physical reactions
  • Priming effect
    • Example study: exposure to a word causes immediate and measurable changes in the ease with which many related words can be evoked.
    • E.g. primed with "EAT", SO_P becomes more easily SOUP but primed with "WASH", SO_P becomes more easily SOAP
    • Priming is not restricted to words
    • Example, Florida effect
      • Priming with words related to aging caused subjects to walk slower
  • Priming is reciprocal (works in both ways), some examples:
    • Walking fast primed for different things/words than walking slow
    • Another examples of reciprocal link: Being amused makes you smile <-> Smiling makes you be more amused
    • Nodding up-down vs. shaking head side-to-side has effect on acceptance/rejection of a message
  • Primes quide out judgments and choices quite much
    • Example study of voting patterns for school funding - whether the voting station was in school or just near had effect on the results
    • Another example study of money priming individualism
    • "Lady MacBeth effect"
    • System 2 believes that it is in charge but System 1 has also effect without us even noticing.
  • System 1 provides impressions that often turn into your beliefs, and is the source of the impulses that often become your choices and your actions.

Cognitive ease

  • Cognitive ease / strain
  • Causes (of cognitive ease) for example: Related experience, clear display, primed idea, good mood
  • Consequences: Feels familiar, feels true, feels good, feels effortless
  • Cognitive ease has multiple causes and they have quite interchangeable effects - it is difficult to tease them apart
  • Illusions of remembering (example of made-up celebrity names)
    • E.g. For a new word, making it easier to see/read -> it will be more likely to have the quality of pastness.
  • Illusions of truth
    • Frequent repetition makes people to believe falsehoods, familiarity is not easily distinguished from truth.
  • How to make a persuasive message
    • Anything you can do to reduce cognitive strain helps: Legibility, simple language, memorable message (rhyming), name easy to pronounce
  • Strain and effort:
    • Reciprocity: Cognitive strain will activate System 1.
    • Study with Shane Frederick's Cognitive Reflection Test: Performance was better with bad font. (Difficulty to read activates System 2)
  • The pleasure of cognitive ease
    • Mind at ease puts a smile on the face
    • Mere exposure effect
  • Ease, mood and intuition
    • Intuition works better when we are on a good mood

Norms, surprises, and causes

  • The main function of System 1 is to maintain and update a model of your personal world, which represents what is normal in it.
  • A capacity for surprises is an essential aspect of our mental life.
  • Norm theory
  • Our mind is eager to see causes and intentions.

A machine for jumping to conclusions

  • System 1 does choices and conclusions automatically and does not keep track of alternatives, or even of the fact that there were alternatives.
  • Daniel Gilbert: For a statement, initial attempt to believe is an automatic operation of System 1. Gilber sees unbelieving as an operation of System 2.
  • Confirmation bias
    • When asked "Is Sam friendly?" different instances of Sam's behaviour will come to mind than would if asked "Is Sam unfriendly"
    • Positive test strategy
  • Halo effect
    • The tendency to like (or dislike) everything about a person - including things you have not observed.
    • The sequence in which we observe characteristics of a person matters how we view that person.
  • To derive the most useful information from multiple sources of evidence, one should try to make these sources independent of each other. (decorrelate error)
  • What you see is all there is (WYSIATI)
    • Associative memory represents only activated ideas.
    • Intuitive thinking is often jumping to conclusions on the basis of limited evidence.

How judgments happen

  • System 2 receives or generates questions - in either case it directs attention and searches memory to find answers
  • System 1 continuously monitors what is going inside and outside the mind and continuously generates "basic assessments" of various aspects of the situation.
    • How things are? Is there a thread or opportunity? ...
    • Example of basic assessment - discriminate friend from a woe at a glance.
    • E.g. Todorov's voting study
  • Sets and prototypes - System 1 deals well with averages but poorly with sums.
  • Intensity matching - System 1's scale of intensity allows matching across diverse dimensions.
    • Example: "Julie read fluently when she was 4 years old." -> "How tall is a man who is as tall as Julie was precocious?"
  • Mental shotgun
    • System 1 carries many computations at any one time + automatically. Other computations are voluntary.
    • The control over intended computations is far from precise - we often compute much more than we need. (This is called mental shotgun)

Answering an easier question.

  • Normally we have intuitive feelings and opinions about almost everything that comes onto our way.
  • Question substitution:
    • The target question is the assessment we intend to produce.
    • The heuristic questions are the simpler questions we answer instead.
  • The mental shotgun makes it easy to generate quick answers to difficult questions without imposing much hard work on the lazy System 2.
  • Intensity matching helps to fit the answers to the original questions
  • Affect heuristics - in which people let their likes and dislikes to determine their view of the world (Paul Slovic)

Thursday, November 7, 2019

Scott Wlaschin - Domain Modeling Made Functional

Notes for Scott Wlaschin book Domain Modeling Made Functional - Tackle Software Complexity with Domain-Driven Design and F#.

Previous DDD books I've read: Domain Driven Design (Eric Evans) & Implementing Domain-Driven Design

As a summary I enjoyed this book. The writing was pretty clear and compact and made me consider F# as an interesting option for building software.

Rough structure of the book

First part of the book was introducing DDD & it's main concepts.

Second part went through types & functions especially in F# context, how to use types to model domain and how to model workflows as pipelines.

Third part discussed first how to implement the domain model, types and pipelines. After that implementation related concepts were discussed: error handling, serialization and persistence.

Fine-grained type system & DDD

F# kind of type system seems to make it quite cheap/easy to introduce lots of types which allows to represent domain on a quite fine-grained way.

Some question marks for me:

  • How difficult it might get with naming if there are very much small types?
  • How much boilerplate code it might require to transfer e.g. an order through a pipeline of types like UnvalidatedOrder, ValidatedOrder, PricedOrder & PricedOrderWithShippingMethod etc.

It would be interesting to try.

Summary of things & practices discussed

DDD

  • Aim to develop a deep shared understanding of the domain
  • Partition the solution space into autonomous, decoupled bounded contexts

DDD with F# kind of language

  • Before implementation, aim to capture the requirements with a type-based notation with both the nouns and the verbs
    • Nouns will usually be represented by an algebraic type system
    • Verbs will usually be represented by functions
  • Aim to capture business rules & constraints in the type system whenever possible.
    • "Make illegal states unpresentable"
  • Aim to design functions pure and total

Summary of FP techniques

  • Compose workflows from smaller functions
  • Parameterize functions when there's a dependency
  • Bake dependencies into a function using partial application
    • Allows to compose functions more easily & hide implementation details
  • Use special functions transforming functions into needed shapes
  • Solve type mismatch problems by lifting types into a common type

Friday, November 1, 2019

SQL Performance Explained

This time I read SQL Performance Explained by Markus Winand. The main theme of the book is database indexing that is probably the most important thing developers should know on SQL database performance.

Note that there is "free web-edition" of the book Use the index, Luke - the web edition has sections matching the book sections (linked to below).

Anyway, I warmly recommend to buy the ebook if you're interested on the subject - It's has a pretty affordable price and has a good bunch of knowledge in pretty small package.

1 - Anatomy of an index

  • Database index is redundant structure referring to the actual information that is stored usually in a different place.
  • Two main data structures:
    • Balanced search tree (B-tree)
    • Doubly linked list
  • Balanced search tree contains leaf nodes.
    • Each leaf node is stored in a database block (smallest storage unit).
    • Each leaf node has index entries sorted.
    • Index entries contain the indexed columns and a reference to the "physical" table row.
  • There is also a doubly linked list connecting the leaf nodes.
  • Index lookup has three steps
    • Tree traversal
    • Following the leaf node chain
    • Fetching the table data from the "physical" table

2 - The Where Clause

  • Execution plan shows how the database actually executes an SQL statement. See DB-specific instructions
    • Created by query optimizer / query planner - usually "cost-based optimizer"
    • Optimization is usually based on statistics
      • Table size (both in rows and blocks)
      • Column level statistics - number of distinct values, range (smallest/largest values), NULL occurrences etc.
      • Index statistics - mainly tree depth
  • Author calls B-Tree traversal first power of indexing.
  • Concatenated indexes
    • Index across multiple columns
    • Note that the order of columns matters - the first column is always usable
  • Full table scan
    • Going through the whole table
    • Note that in some cases that can be the most efficient operation
      • DB might be able to read larger chunks at a time
      • No need for "random access"
    • Note: It is important how many rows are got from the index for filtering
  • Function-based indexing
    • Summa summarum has some details to be aware of!
    • By default aim to index the original data - usually that is the most useful info
  • Parameterized queries
    • Queries with bind parameters (also referred to as dynamic parameters or bind variables)
    • DBs with execution plan cache can reuse execution plan when using bind parameters
    • When using bind parameters, the optimizer has no concrete values to determine their frequency -> will always select the same execution plan
      • There are not too many cases where that affects the execution plan
  • Searching for ranges
    • Can utilize indexes
    • As a basic rule, keep the scanned range as small as possible.
    • As a rule of thumb, index first for equality and then for ranges.
  • LIKE expressions
    • can only use the characters before the first wildcard.
      • -> Avoid LIKE expressions with leading wildcard
    • Be aware that with bind parameters the DB does not know about the leading wildcard
      • Work-around append hack: Instead of (foo LIKE ?) use (foo || '' LIKE ?). (Use only as a last resort)
  • Index merge
    • In most cases one index with multiple columns is better than multiple single-column indexes
    • Exception: Queries with multiple range conditions (E.g. FOO < ? AND BAR < ?)
    • Methods for combining indexes
      • Index join
      • Bitmap indexes (DWH, mainly not suitable for OLTP)
  • Partial indexes
    • Index only part of the rows
      • E.g. CREATE INDEXES foo_bar ON foo (bar) WHERE baz = 123
    • Very common in queuing systems where most of processing is for unprocessed entries.
  • Oracle NULL has some surprises, e.g.:
    • Oracle treats an empty string quite much as NULL
    • As a gotcha: Oracle does not include rows in an index if all indexed columns are NULL.
    • Can be used to emulate partial indexes.
  • Obfuscating conditions - where clause patterns that prevent index usage
    • Watch out indexing with DATE types - Summa summarum: Write queries for continuous periods as explicit range conditions.
    • Numeric strings - Summa summarum: Use numeric types for numbers.
    • Combining columns - Summa summarum: When combining multiple columns in a range condition, use a redundant condition.
    • ...

3 - Performance and Scalability

  • Environmental parameters that affect performance
    • Data volume
    • System load
    • Hardware
  • Scalability - Impacts of data volume
    • Indexing
    • Pay attention to predicate information of the query plans
    • Pay attention to filter predicates
  • Scalability - Impacts of system load
    • Bigger hardware is not always faster but it can usually handle more load.
      • Same with scaling horizontally (more servers)

4 - The Join Operation

Three approaches:

  • Nested loops - Kind of server-side N+1 selects
  • Hash join - Query candidate records from another table into a hash table
    • This is the usually used one.
  • Sort merge - Kind of a zipper
    • Works well if the inputs are already sorted

5 - Clustering Data

  • "Clustering" here: Storing data that is accessed together close to each other to make data access require fewer IO operations.
  • The author refers to clustering as the second power of indexing.
  • Index predicates
    • Having a filtered value in the index can reduce table access.
    • Table access is not so big deal if accessed rows are stored in same table block. (All rows will be read together in one read operation)
    • Index clustering factor - Correlation between order of row entries in the index and order of rows in the table.
    • As a tip, don't introduce a new index for filter predicates but extend an existing index.
  • Index-only scan
    • Performance difference depends on the number of accessed rows and index clustering factor.
  • As a special case, some databases support index-organized tables
    • That is, a B-tree without a heap table
    • Note that secondary indexes come very inefficient to use.

6 - Sorting and Grouping

  • Indexed ORDER BY execution
    • Saves sorting effort
    • Allows pipelined execution - returning first results without processing the all data.
    • -> The author calls pipelined order by third power of indexing
  • Remember that databases can read indexes in both directions.
  • GROUP BY - two different approaches
    • Hash approach - Use a temporary hash table
    • Sort/group approach - Sort the data first by the grouping key
      • Can use index to avoid sorting -> enables pipelined GROUP BY

7 - Partial Results

  • E.g. querying for top N rows.
  • Pipelined ORDER BY is very powerful to optimize these.
    • -> DB can optimize for partial result only if it knows it when planning the query
    • -> Inform the DB when you don't need all rows.
  • Paging
    • Deterministic sort order is needed
    • Two approaches
      • Offset method - Number of rows from the beginning
        • Vulnerable to drifting
        • Gets slower when going further.
      • Seek method - Search for last entry of previous page

8 - Insert, Delete and Update

  • Avoid redundant indexes whenever possible
  • Note that the first index makes the greatest difference
  • With INSERT & DELETE, each index adds time spent.
  • With UPDATE - indexes on columns that are not updated don't generally add so much time.

Bonus

The author of the book has also a nice website modern-sql.com on interesting newer SQL features.

Thursday, May 9, 2019

Designing Data-Intensive Applications

After some pauses in the middle, I finally finished Martin Kleppmann's book Designing Data-Intensive Applications.

In short, it's a great book on different aspects of data: storing data, transferring data, processing data, distributed systems etc. etc.
The writer was focusing on concepts rather than details of individual tools. Also he had a nice balance of theory and practise.

Each chapter has nice old-school map as TOC. For the total TOC, check out the cool poster.

Friday, September 28, 2018

Implementing Domain-Driven Design (Book notes)

2018-implementing-ddd

Notes from Vaughn Vernon book Implementing Domain-Driven Design

Note that I have read the "Blue Book" (Eric Evans' Domain-Driven Design) previously, so mainly I haven't repeated things from that book.

Foreword and Preface

Foreword by Eric Evans

  • Nine years after my book (...) was published, there's actually a lot to say about DDD that is new, and there are new ways to talk about the fundamentals.

Preface

  • Sometimes DDD is first embraced as a technical tool set (referred to as DDD-Lite by some).
  • If there is a single invention Evans delivers to the software development community, it is the Ubiquitous language.

Chapter 1: Getting Started with DDD

  • DDD isn't first and foremost about technologies. In its most central principles, it's about discussion, listening, understanding, discovery and business value, all in an effort to centralize knowledge.
  • If you're capable of understanding the business ..., you can at a minimum participate on discovery process for a Ubiquitous Language.
  • How DDD helps?
    • 1: DDD brings domain experts and software developers together in order to develop software that reflects the mental model of the business experts
      • Instead of "most realistic", thrive to deliver a model that is most useful for the business.
    • 2: DDD addresses the strategic initiatives of the business
    • 3: DDD has tactical design modeling tools to analyze and develop software
  • Use DDD to simplify, not to complicate!
  • Business value for DDD
    • The organizations gains a useful model of its domain
    • A refined, precise definition and understanding of the business is developed
    • ...

Chapter 2: Domains, Subdomains, and Bounded Contexts

These things are "Strategic design".

  • In broad sense, a domain is what an organization does and the world it does it in.
  • Bounded Context is chiefly a linguistic boundary.
    • Some projects fall into the trap of attempting to create an all-inclusive model. (Which is a pitfall)
    • Same term might have very different meaning in different bounded contexts.
    • Bounded Context often marks off a system, an application or a business service.
    • With persistence, a database schema will live inside the boundary.
    • How big should it be? A bounded context should be as big as it needs to be in order to fully express it's complete Ubiquitous Language.
    • Often a single team for a single bounded context works pretty well

Division between different parts of the business domain:

  • Core Domain is a part that is of primary importance to the success of the organization.
    • Supporting Subdomain: Bounded Context that models some aspect of the business that is essential, yet not Core.
    • Generic Subdomain: Subdomain that captures nothing special to the business, yet is required for the overall business solution.

Problem space vs solution space. (This chapter deals mainly on problem space assessment)

Chapter 3: Context Maps

This chapter focuses on the solution space assessment.

  • Context map shows the mappings between Bounded Contexts.
  • By drawing a Context Map, you will be forced to think about your relationships with all other projects / Contexts you depend on.
  • A Context Map should capture the existing terrain, not the imagined future. First focus on the current situation and where you are. After that determine where to go next.
  • (Going through the patterns in Evans' "Strategic Design" part)
  • We might have state dependent from another system.
    • Instead of caching whole dependent objects, we create local domain objects translated from the foreign model.

Chapter 4: Architecture

  • The goal is to use just the right choices and combinations of architecture and architecture patterns.

Layers

Hexagonal / Ports and Adapters

  • See Alistair Cockburn's Hexagonal Architecture article.
  • Ports and Adapters name also used.
  • Instead of "Front end" and "Back end" Hexagonal looks are "Inside" and "Outside"
    • Domain Model furthest inside.
  • Normally we don't implement the ports ourselves. (Most of that comes from a framework, container etc.)
  • For remainder of the chapter, assume that Ports and Adapters approach is used.

Service-Oriented

REST

  • Two alternatives for combining DDD and RESTful HTTP
    • 1: Create a separate Bounded Context for the system's interface layer
    • 2: Reflect domain objects in resources
      • As a problem, any changes to objects structure are immediately reflected in remote interfaces.

CQRS

From Wikipedia:

... every method should either be a command that performs an action, or a query that returns data to the caller, but not both. In other words, asking a question should not change the answer. More formally, methods should return a value only if they are referentially transparent and hence possess no side effects.

Examining Areas of CQRS (see overview image)

  • Query Processor takes queries from client.
  • Query Model (aka Read Model)
  • Command Processors
  • Command Model (aka Write Model)
  • Event Subscriber updates the Query Model
    • Synchronous or asynchronous? Depends on the normal load of the system, where the query model DB is stored, data consistency constraints and performance requirements.
    • If Query model is Eventually Consistent, it needs to be considered in the UI. Different approaches:
      • As a trick, show the data UI in the UI as the command would already be successfully executed.
      • Show date and time from the query model that the user is currently viewing.

Summa summarum, CQRS introduces a number of competing forces

Event-Driven Architecture

  • Pipes and Filters
  • Long-Running processes (aka Sagas)
    • Problem of knowing which process was ending etc?
      • One way is to assign a unique Process identity.
      • Store an aggregate-like state object tracking process state & completion.

Event Sourcing

  • Sometimes the business cares about tracking changes that occur to the objects in the domain model.
  • -> Store history of everything that's happened.
  • To avoid playback of many events, we can apply optimization with (Aggregate) state snapshots.

Data Fabric and Grid-Based Distributed Computing

Chapter 5: Entities

  • Developers have tendency to focus on data rather than the domain.
    • One reason being many approaches placing importance on the database.
    • Instead of designing domain concepts with rich behavior, we might think primarily about the attributes
  • Domain concept is designed as an Entity when we care about its individuality.
    • Distinguishing it from all other objects in a system is a mandatory constraint.
  • Some strategies for identity creation:
    • User provides one or more unique values as input to the application.
    • Application internally generates an identity using some algorithm that ensures uniqueness
    • The application relies on a persistence store, e.g. database.
    • Unique identity is already determined by another Bounded Context.
  • Identity generation timing can be done either
    • Early (identity is generated before the Entity is persisted. As part of object's construction.
    • Late (identity is generated when the Entity is persisted). As part of object's persistence.
  • Surrogate identity
    • Some tools (e.g. ORMs) want to deal with Object identity on their own terms.
    • -> Two identities:
      • One designed for the domain model
      • One for the tool, known as a surrogate identity
    • It's best to hide the surrogate attribute from the outside world.
  • Entity validation - can be done at three levels
    • Validating single attributes/properties (not null, certain values, certain format, ...)
      • Referred to as defensive programming.
    • Validating Whole Objects
      • Validations that need to have access to the state of the entire object.
      • Specification/Strategy pattern might be used.
    • Validating Object Compositions
      • Often at Aggregate level.

Chapter 6: Value Objects

  • Evans: When you care only about the attributes of an element of the model, classify it as a Value Object.
    • Treat the Value Object is immutable.
    • Don't give it any identity.
  • We should thrive to model using Value Objects instead of Entities whenever possible.
  • Make sure you address the Ubiquitous Language.
  • Value objects usually possess most of these characteristics:
    • It measures, quantifies or describes a thing in the domain.
    • It can be maintained as immutable.
    • It models a conceptual whole.
      • Parent reference to a Value Object is not just an attribute. Rather, it is a property of the containing parent object/thing...
    • It is completely replaceable.
    • It can be compared with others using Value equality.
    • It supplies its collaborators with Side-Effect-Free Behavior.
  • Standard Types Expressed as Values.
    • E.g. Java enums can be used as State object
  • Vernon prefers naming valuePercentage() over getValuePercentage()
    • getValuePercentage() is a technical statement
    • valuePercentage() is a fluent human-readable language expression.
  • If at all possible, design your data model for the sake of your domain model, not vice versa.

ORM things discussed:

  • ORM and Single Value Objects
    • Often Value Object is denormalized into its parent Entity's DB row.
  • ORM and Many values serialized into a single column
    • e.g. List/Set
    • Potential drawbacks to consider
      • Column width
      • Querying - Individual value elements not queryable.
      • Requires custom user type
  • ORM and Many values backed by a database entity
  • ORM and Many values backed by a join table
    • Downsides
      • join needed
      • nulls not supported
      • value type mapped may itself not contain a collection
    • Generally to be avoided
  • ORM and Enum-as-State Objects
    • Requires custom user type.

Chapter 7: Services

  • A Service in the domain is a stateless operation that fulfills a domain-specific task.
    • Often the best indication for service being needed is when the operation you need to perform doesn't fit either to an Aggregate of a Value Object.
  • Don't confuse a Domain Service with an Application Service.
    • We don't want to house business logic in an Application Service.
    • Application Service would normally be a client of a Domain Service.
    • Transactions and security should be addressed as application concerns in Application Services, not in Domain services.
  • When would an operation not belong on an Entity of Value Object? You can use a Domain Service to
    • Perform a significant business process.
    • Transform a domain object from one composition to another.
    • Calculate a Value requiring input from more than one domain object.
  • Make sure you need a service.
  • Is Separated Interface necessary?
    • May be a more a matter of style in cases where the service is always domain specific and will not have technical implementation or multiple implementations.

Chapter 8: Domain Events

  • Contemporary definition for Domain Events: Something happened that domain experts care about.
    • Although domain experts might not initially be aware of the need for every kind of Event, they should understand the reasons for them, as they are included in discussions.
  • When Events are delivered to interesting parties (either local of foreign systems), they are generally used to facilitate eventual consistency
    • This is purposeful and by design.
    • Can eliminate the need for two-phase commits & support of the rules of Aggregates.
  • Does every Aggregate command result in an Event?
    • It is important also to know when to disregard happenings in the domain that experts or the business, as a whole don't care about.
    • Because of technical implementation aspects it is possible that Event will be more prolific than domain experts care about.
  • Modeling Events
    • Name Events and their properties according to the Ubiquitous Language.
    • Typical naming
      • Command operation: BacklogItem#commitTo(Sprint sprint)
      • Event outcome: BacklogItemCommitted
    • At times Events are created by direct request from clients and don't fit a single Aggregate.
      • When that happens, the Event can be modeled as an Aggregate and retained in its own Repository.
    • Identity
      • At times it may be necessary to distinguish Events on from another, but the need may be rare.
      • Unique identity may be necessary when Events are published outside the local Bounded Context, forwarded by messaging infrastructure.
  • Publishing Events
    • Publisher
      • Typical use of Domain Events is when an Aggregate creates an Event and publishes it.
      • The publisher resides in a Module of the model, but it doesn't model an aspect of the domain.
    • Subscribers
      • Generally Application Services subscribe to Domain Events. Sometimes Domain Services.
      • One thing the subscriber should not do is get another Aggregate instance and execute modifying command on it.
      • This would violate the modify-single-aggregate-instance-in-single-transaction rule of thumb.
    • Events are domain-wide concept, not just a concept in a single Bounded Context
      • Events are published to any number of Bounded Contexts of other Subdomains.
  • Spreading the News to Remote Bounded Contexts
    • Some form of messaging takes place.
    • Commitment to Eventual consistency is needed
      • The changes in one model that influence one or more other models will not be fully consistent for some period of time.
    • (At least) two mechanisms must be consistent with each other:
      • Persistence store for the domain model
      • Persistence store backing the messaging infrastructure
    • Autonomous Services and Systems
      • A high degree of independency from other systems is achieved by avoiding in-band RPCs.
      • Use asynchronous messaging to achieve a greater degree of independence between systems - autonomy.
    • Latency Tolerances
      • It may surprise developers to learn that most times, several seconds, minutes, hours or even days between consistent states might be completely tolerable.
      • Not true for all cases.
      • We must not assume that near-consistent time frames would be always imperative for any given domain.
  • Event Store
    • Consider what you could to if you were to store a discrete event for every model command:
        1. Use the Event Store as a queue for publishing all Domain Events through a messaging infrastructure.
        1. Use the same Event Store to feed REST-based Event notifications to polling clients
        • Logically the same as point 1, but different in actual us.
        1. Examine a historical record of the result of every command ever executed on the model.
        1. Use the data in business analytics (trending, forecasting etc.)
        1. Use the Events to reconstitute each Aggregate instance when it is retrieved from its Repository.
        1. Undo blocks of changes to an Aggregate
  • Architectural Styles for Forwarding Stored Events
    • Publishing Notifications as RESTful Resources
      • (+) If potentially many clients can go to a single well-known URI for the same set of notifications, the RESTful approach works well.
      • (-) If one or few consumers are required to pull from multiple producers for resources in order to get a single set of tasks to performed in a specific sequence, the RESTful approach won't probably work too well.
    • Publishing Notifications through Messaging Middleware
  • Event De-Duplication
    • One way to deal with the possibility of duplicate message delivery is for subscriber model operation to be idempotent.
    • When idempotency is not a viable option, you can instead design the subscriber/receiver itself to be idempotent.
      • Messaging product might support this off-the-shelf.

Chapter 9: Modules

  • In a DDD context, Modules in your model serve as named containers for domain classes that are highly cohesive with one another.
  • Simple Do/Don't Rules for Module Design
    • Do design Modules to fit modeling concepts.
    • Do name Modules per the Ubiquitous Language.
    • Don't create Modules mechanically according to a general component type or pattern being used.
    • Do design loosely coupled Modules.
    • Do strive for acyclic dependencies on peer Modules when coupling is necessary.
    • Do relax the rules a bit between child and parent Modules.
    • Don't make Modules as a static concept of the model, but allow them to be molded along with the objects that they organize.
  • Module naming conventions
    • One approach is to divide model and services, e.g.
      • com.foobar.identityaccess.domain.model
      • com.foobar.identityaccess.domain.service
      • com.foobar.collabaration.domain.model
      • com.foobar.collabaration.domain.service
      • Fits well with that we're designing and implementing a model of a domain, not domain.
    • Another approach is to drop model/service division
      • com.foobar.collabaration.domain.conceptname
  • Modules in other layers
    • Same structure can be followed at other layers
      • RESTful resources e.g. com.foobar.collabaration.resources, com.foobar.collabaration.resources.view
      • Application layer modules, e.g. com.foobar.agilepm.application.team

Chapter 10: Aggregates

  • As a start: What is an Aggregate? ... What is this concept of invariants and a consistency boundary all about?
    • The last question is an especially relevant one.
  • Rule: Model True Invariants in Consistency Boundaries
    • An invariant is a business rule that must always be consistent.
    • Different kinds of consistency: Transactional consistency and eventual consistency.
      • When discussing invariants, we're referring to transactional consistency.
    • Consistency boundary logically asserts that everything inside adheres to a specific set of business invariant rules.
    • With a typical persistence solution, we use a single transaction to manage consistency.
    • A properly designed Aggregate is one that can be modified in any way required by the business with its invariants completely consistent within a single transaction.
    • A properly designed Bounded Context modifies only one Aggregate instance per transaction in all cases.
    • -> We can't correctly reason on Aggregate design without applying transactional analysis.
    • Aggregates are chiefly about consistency boundaries and not driven by a desire to design object graphs.
    • Note that these rules are rules of thumb.
  • Rule: Design Small Aggregates
    • With big aggregates, multiple large collections etc. might be loaded during many simple operations.
    • What does "small" mean? The correct minimum is however many are necessary, and no more. :)
      • Often target for just the Root Entity and a minimal number of attributes and/or Value-typed properties.
    • Don't Trust Every Use Case
      • A new use case may lead to insights that push us to remodel the Aggregate.
      • Be skeptical here, too.
      • A large-cluster Aggregate might be problematic
      • Often, in such cases, the business goal can be achieved with eventual consistency between Aggregates.
  • Rule: Reference Other Aggregates by Identity
  • Rule: Use Eventual Consistency Outside the Boundary
    • If executing a command on one Aggregate instance requires that additional business rules execute on one or more other Aggregates, use eventual consistency.
    • "Ask Whose Job It Is" principle:
      • When examining the use case, ask whether it's the job of the user executing the use case to make the data consistent.
        • If it is, try to make it transactionally consistent.
        • If it is another user's job, or the job of the system, allow it to be eventually consistent.
  • Reasons to break the rules
    • 1: User Interface Convenience
    • 2: Lack of Technical Mechanisms
      • user-aggregate affinity: Are the business workflows such that only one user would be focused on one set of Aggregate instances at any given time?
    • 3: Global transaction
    • 4: Query performance
  • Implementation
    • Root Entity with Unique Identity
    • Law of Demeter: Any given method on any object may invoke methods only on the following:
      • Itself
      • Any parameters passed to it
      • Any object it instantiates
      • Self-contained part objects that it can directly access

Chapter 11: Factories

  • A Factory may or may not have additional responsibilities in the domain model other than object creation.
  • Two main options:
    • Factory on Aggregates
      • Factory Methods on Aggregates allow you to express the Ubiquitous Language in ways not possible through constructors alone.
    • Factory on Service

Chapter 12: Repositories

Evans:

For each type of object that requires global access, create an object that can provide an illusion of an in-memory collection of all objects of that type. ... Provide methods to add and remove objects, which will encapsulate the actual insertion or removal of data in the data store. ... Provide Repositories only for Aggregate roots

  • Repositories are all about persistency.
  • Generally, each persistent Aggregate type will have a Repository
  • There is also option of using persistence mechanism's Session or Unit of Work as such. Generally to be avoided.
  • Two kinds of Repository design: collection-oriented design and persistence-oriented design
  • Collection-Oriented Repositories
    • Mimic a collection.
    • A repository interface that does not hint that there is an underlying persistence mechanism, avoiding any notion of saving/persisting data to a store.
    • For a Java in-memory collection, there is no need to do anything special to get the collection to recognize the changes to the objects it contains.
    • Take-aways:
      • A repository should mimic a Set interface.
      • You must not allow instances of the same object to be added twice.
      • When retrieving objects from a Repository and modifying them, you don't need to "re-save" them to the Repository.
    • The persistence mechanism must in some way support tracking changes. Can be done in various ways, including the following:
      • Implicit Copy-on-Read - Each persistent object is copied when it is loaded from the data store.
      • Implicit Copy-on-Write - Proxy objects that make a copy of the managed object when required.
    • Note that in general it is possible that instances of some Aggregate types must never be removed through normal application use cases.
      • Instances might be needed after they are no longer usable in the application. For e.g. referential and/or historical purposes.
      • In this kind of cases Aggregate instances can be marked e.g. as disabled, unusable, or in some other way logically removed.
    • Repository interface is often in domain.
      • Implementation can be placed in infrastructure layer.
      • Another approach is to put implementation in a submodule under Aggregate and Repository module
    • Exceptions: Since we're going to abstract away the implementation details in general, we want to isolate clients from those also with exceptions.
    • TopLInk has both Session and Unit of Work.
      • With Unit of Work, objects to be changed are explicitly registered to the Unit of Work. -> More efficient use of memory & processing power, not so transparent abstraction
  • Persistence-Oriented Repositories
    • "Save-based" approach.
    • The case when the persistence mechanism doesn't implicitly or explicitly detect and track changes.
    • Every time you create a new Aggregate instance or change a pre-existing one, you have to put it explicitly into the data store by using e.g. save.
    • Repository operation both when aggregates are created and when they are modified. (In comparison with collection-oriented repository no explicit operation when existing aggregate is updated)
    • Take-aways:
      • We must explicitly put() both new and changed objects into the store, replacing value previously associated with the given key.
      • These data stores are sometimes called Aggregate Stores or Aggregate-Oriented Databases.
  • Additional behavior
    • Sometimes it is beneficial to provide additional behavior on a Repository interface.
    • E.g. calculations that must be performed in the data store in order to meet some non-functional requirement.
    • At times it may be advantageous to query Aggregate parts out of the Repository without directly accessing the Root itself.
      • This should be used primarily to address performance concerns.
      • Use with caution
      • Special finder methods
      • use case optimal query (often resulting as a Value object)
    • If there's need for many finder methods supporting use case optional queries on multiple Repositories, it's probably a code smell
      • Code smell named Repository masks Aggregate mis-design
  • Transactions
    • Domain layer is never the correct place to manage transactions. Usually transactions should be managed in the Application Layer.
    • Repository implementations should have access to the same Session / Unit of Work for the transaction the Application Layers started.
    • Be careful not to overuse the ability to commit modifications to multiple Aggregates in a single transaction. (If need be, revisit Aggregates)
  • Repository vs Data Access Object (DAO)
    • DAO is expressed in terms of database tables, providing CRUD operations
  • Testing repositories: Two ways to look at testing Repositories
    • Repositories themselves have to be tested.
    • Code using Repositories has to be tested.

Chapter 13: Integrating Bounded Contexts

  • There are always multiple Bounded Contexts in any project of significance, and there will be need to integrate.
  • A few reasonably straightforward ways to integrate Bounded Contexts
    • Expose an API in another Bounded Context and use that API via RPCs from another.
    • Use of messaging mechanism
    • RESTful HTTP
  • Distributed systems are fundamentally different. Author's Principles of Distributed Computing
    • The network is not reliable.
    • There is always some latency, and maybe a lot.
    • Bandwidth is not infinite.
    • Do not assume that the network is secure.
    • Network topology changes.
    • Knowledge and policies are spread across multiple administrators.
    • Network transport has cost.
    • The network is heterogeneous.
  • Experience with
    • Open Host Service: Enhance and expand the protocol to handle new integration requirements
    • -> Provide only what integrators need at present, based on a range of use case scenarios.
  • If at all possible, it is best to minimize or even completely eliminate information duplication across Bounded Contexts.
    • It may not be possible to do that entirely.
    • E.g. SLAs may make it impractical to retrieve remote data every time it is needed.
    • Having the goal to reduce the amount of foreign information will make our jobs much easier.

Chapter 14: Application

  • A domain model often lives at the heart of an /application/.

  • Here we're using the term /application/ somewhat interchangeably with /system/ and /business service/.

    • System is usually a solution with many applications.
  • UI

    • How do we render domain objects onto the glass?
      • There's controversy and disagreement on how best to render objects of the domain model onto the user interface.
      • The UI often benefits from views of data richer than is required to accomplish the direct task.
      • UI often needs to render properties of multiple Aggregate instances.
      • In most use cases, the user performs a state-mutating task that is applied to just one Aggregate instance.
    • DTO approach
      • Design a DTO to hold the entire number of attributes that are needed to display the view.
      • DTO pattern was originally designed to deal with a remote presentation tier that consumes the DTO instances.
    • Mediator to publish aggregate internal state
      • To reduce coupling between the model and it's (UI) clients
      • e.g. BacklogItem.provideBacklogItemInterest(BacklogItemInterest interest) that would call interest.informTenantId(...) etc.
    • Domain Payload Object (DPO)
      • A variant of DTO within a single virtual machine.
      • Contains a reference to whole Aggregate instances.
    • State Representations of Aggregate instances
      • REST-based resources
      • It's good to think a set of RESTful resources as a separate model (View Model / Presentational Model) to avoid coupling clients straight to the domain model.
    • Use Case Optimal Repository Queries
      • Creating a DTO/PTO straight with Repository.
      • This approach has similar motivations than CQRS. (Might be worth going all the CQRS route)
    • Dealing with Multiple Disparate Clients
      • You may design your Application Services to accept Data Transformer
    • Rendition Adapters and Handling User Edits
      • In whatever way your domain data is provided from Application Services - through DTOs, DPOs, or state representations - and whatever presentation framework you use, you may be able to benefit from Presentational Model.
  • Application Services

    • Application Services are direct clients of the domain model. (Options on the logical location were introduced earlier in Architecture chapter)
    • When using an ACID database, the Application Services also control transactions.
    • Security is also commonly cared for by Application Services.
    • Example
      • Some types from the domain model are used in Application Service method signatures.
      • -> UI needs to be aware of these types and depend on them.
      • Alternatively, signature with only primitive types, DTOs and/or Command objects could be used.
      • Spring @Transactional used
      • Potentially also security concerns.
  • Infrastructure

    • To provide technical capabilities for other parts of the application.

Appendix: Aggregates and Event Sourcing: A+ES

Note:

Actual content:

  • Basics
    • Event Sourcing can be used to represent the entire state of an Aggregate as a sequence of Events that have occurred since it was created.
    • State of the Aggregate can be rebuilt from the events.
  • Example flow of operations:
      1. Client calls Application Service
      1. Loads Aggregate state from Event Stream
      1. Call Aggregate method passing parameters got, along with Domain Services as needed
      1. Commit published Events as a Unit of Work to the Event Store to persist the state of the aggregate
      1. New Events are published from Event Store to all subscribers
  • Command Handlers
    • Control the task management of our application.
    • Command Handler effectively replaces the Application Service method.
    • Decoupling the client from the Service can /enhance load balancing, enable competing consumers, and support system partitioning./
    • This approach creates /temporal decoupling/ between clients and the Application Service.
  • Performance
    • Some patterns to apply:
      • Cache Event Streams in server memory (as Events are immutable once written to the Event Stream)
      • Use /snapshots/ of Aggregate instances to avoid loading and replaying big amounts of an Event Stream.
    • With A+ES Aggregates, Aggregates can be partitioned among multiple processes or machines by Aggregate Identity.
  • A+ES aggregates tend to be smaller (following Aggregate Rules of Thumb)
    • Because creating new Aggregates tends to be easier with A+ES than traditional aggregate persistence.
  • Read Model Projections
    • One of the common concerts of A+ES: How to query the Aggregates by their properties?
    • Here Read Model Projections can help
      • Domain Event subscribers that project Events to a persistent Read Model.
      • See sample lokad-cqrs
  • Event enrichers
    • One of A+ES problems comes from event's dual purpose.
      • i) Aggregate persistence
      • ii) Communicate domain-level happenings around the enterprise.
    • Subscribers (ii) often need additional information.
    • We can simplify this by enriching the domain events with additional data.
      • Not essential for reconstructing the aggregate but simplifies Event subscribers