Stochastic logistic models reproduce experimental time series of microbial communities

1. LanaDescheemaeker12
2. SophiedeBuyl[email protected]12

1. Applied Physics Research Group, Physics Department, Vrije Universiteit Brussel
   BrusselBelgium
2. Interuniversity Institute of Bioinformatics in Brussels, Vrije Universiteit Brussel Université Libre de Bruxelles
   BrusselsBelgium

• Research Article

• Computational and Systems Biology

• stochastic generalized lotka-volterra equations
• logistic model
• microbial communities dynamics
• noise analysis
• None

• publisher-id55650
• doi10.7554/eLife.55650
• elocation-ide55650

#general imports

# Data manipulation
import pandas as pd
import numpy as np

# Options for pandas
pd.options.display.max_columns = 50
pd.options.display.max_rows = 30

from IPython import get_ipython
ipython = get_ipython()

# autoreload extension
if 'autoreload' not in ipython.extension_manager.loaded:
    %load_ext autoreload

%autoreload 2

import matplotlib.pyplot as plt
from matplotlib import gridspec
%matplotlib inline

import time
np.random.seed(int(time.time()))


#specific imports

import matplotlib as mpl
from noise_analysis import noise_color
from scipy import stats
from noise_properties_plotting import noise_cmap_ww, noise_lim, PiecewiseNormalize, \
    PlotTimeseriesComparison, PlotNoiseColorComparison
from generate_timeseries import Timeseries
from noise_parameters import NOISE, MODEL

from matplotlib import font_manager
font_manager._rebuild()

from elife_settings import set_elife_settings, ELIFE

set_elife_settings()

Introduction

Microbial communities are found everywhere on earth, from oceans and soils to gastrointestinal tracts of animals, and play a key role in shaping ecological systems. Because of their importance for our health, human-associated microbial communities have recently received a lot of attention. According to the latest estimates, for each human cell in our body, we count one microbe 27Sender et al., 2016. Dysbiosis in the gut microbiome is associated with many diseases from obesity, chronic inflammatory diseases, some types of cancer to autism spectrum disorder 12Gilbert et al., 2016. It is therefore crucial to recognize what a healthy composition is, and if unbalanced, be able to shift the composition to a healthy state. This asks for an understanding of the ecological processes shaping the community and dynamical modeling.

The dynamics of complex ecosystems can be studied by considering the number of individuals of each species, referred to as abundances, at subsequent time points. There are several ways to characterize experimental time series properties. Models typically focus on one specific aspect such as the stability of the community 22May, 19723Coyte et al., 201517Levine et al., 201714Grilli et al., 20179Gavina et al., 201810Gibbs et al., 2018, the neutrality 8Fisher and Mehta, 201433Washburne et al., 2016, or mechanisms leading to long-tailed rank abundance distributions 30Solé et al., 20021Brown et al., 200224McGill et al., 200721Matthews and Whittaker, 2015. Different types of dynamical models have been proposed. A first distinction can be made between neutral and non-neutral models. Neutral models assume that species are ecologically equivalent and that all variation between species is caused by randomness. In such models, no competitive or other interactions are included. A second distinction is made between population-level and individual-based models. Generalized Lotka-Volterra (gLV) models describe the system at the population level and assume that the interactions between species dictate the community’s time evolution. Both deterministic and stochastic implementations exist for gLV models. Stochastic models include a noise term. There are multiple origins of the noise: intrinsic noise captures the fluctuations due to small numbers, extrinsic noise models external factors such as changing immigration rates of species or changing growth rates mediated by a varying flux of nutrients. Individual-based or agent-based models include self-organized instability models 30Solé et al., 2002 and the controversial neutral model of 16Hubbell, 2001; 26Rosindell et al., 2011. A classification scheme that assesses the relative importance of different ecological processes from time series has been proposed in 7Faust et al., 2018. The scheme is based on a test for temporal structure in the time series via an analysis of the noise color and neutrality. Applied to the time series of human stool microbiota, it tells us that stochastic gLV or self-organized instability models are more realistic. Here, we will however only focus on stochastic gLV models. The reason for this is twofold. First, one can encompass the whole spectrum of ecosystems from neutral to niche with gLV models 8Fisher and Mehta, 2014. Second, we aim at describing dense ecosystems and even though an individual-based model might be more accurate, in the large number limit it will be captured by a Langevin approximation, that is, by the stochastic gLV model.

Our goal is to compare time series generated by stochastic gLV models with experimental time series of microbial communities. We aim at capturing all observed properties mentioned above—the rank abundance profile, the noise color, and the niche character—as well as the statistical properties of the differences between abundances at successive time points with one model. As is shown in Properties of experimental time series, the abundance distribution is heavy-tailed, which means that few species are highly abundant and many species have low abundances. Despite the large differences in abundances, the ratios of abundances at successive time points and the noise color are independent of these abundances and although the fluctuations are large, the results of the neutrality tests indicate that the experimental time series are in the niche regime. To sum up, we seek growth rates, interaction matrices, immigration rates, and an implementation of the noise in stochastic gLV models to obtain the experimental characteristics.

We simulated time series using gLV equations. The interaction matrices are random as was introduced by 22May, 1972. The growth rates are determined by the choice of the steady-state, which is set to either equal abundances for all species or abundances according to the rank abundance profiles found for experimental data. For the noise, we consider different implementations corresponding to different sources of intrinsic and extrinsic noise.

Our analysis constrains the type of stochastic gLV models able to grasp the properties of experimental time series. First, we show that there is a correlation between the noise color and the product of the mean abundance and the self-interaction of a species. The noise color profile for such models will, therefore, depend on the steady-state. This implies that imposing equal self-interaction strengths for all species, what can be done to ensure stability 8Fisher and Mehta, 201411Gibson et al., 2016, is incompatible with the properties of experimental time series. Second, from the differences between abundances at successive time points, we conclude that a model with mostly extrinsic (linear) noise agrees best with the experimental time series. Third, neutrality tests often result in the niche regime for time series generated by noninteracting species with noise. We, therefore, conclude that all stochastic properties of experimental time series are captured by a logistic model with large linear noise. However, interactions are not incompatible with those properties. This suggests using stochastic logistic models as null models to test for interactions. Our results go along the lines of the ones obtained by 15Grilli, 2019 which state that the stochastic logistic model can be interpreted as an effective model capturing the statistics of individual species fluctuations.

All codes are available online (see Additional files: Code).

Results

Properties of experimental time series

We study time series from different microbial systems: the human gut microbiome 4David et al., 2014, marine plankton 20Martin-Platero et al., 2018, and diverse body sites (hand palm, tongue, fecal) (2Caporaso et al., 2011; Figure 1A). A study of the different characteristics for a selection of these data is represented in Figure 1. The complete study of all time series can be found in Supplementary file 1: Analysis of experimental data. We propose a detailed description of the properties of the experimental time series. They fall essentially into two categories. The stability and rank abundance are tightly connected to the deterministic part of the equations while the differences between abundances at successive time points and noise color explain the stochastic behavior. The neutrality is more subtle and depends on the complete system.

# Load dataframes with experimental data

# MartinPlatero plankton data

df_ts = {}

path = 'Data/MartinPlatero/'
files = ['41467_2017_2571_MOESM5_ESM_MartinPlatero_Plankton_Eukarya.csv']
    #['41467_2017_2571_MOESM4_ESM_MartinPlatero_Plankton_Bacteria.csv']
keys = ['plankton_eukarya']
    #['plankton_bacteria'] 

for i, f in enumerate(files):
    x = pd.read_csv(path+f, na_values='NAN', index_col=0)
    x = x.iloc[:, :-1] # delete last columns which contains details on the otu's
    
    # only keep 200 most abundant species
    sum_x = x.sum(axis='columns')
    
    x = x[sum_x >= np.sort(sum_x)[-200]]
    
    x = x.T # species are in rows instead of columns
    
    x.insert(0, 'time', [int(j[4:7]) for j in x.index]) # day
    
    x = x.groupby('time').agg('mean').reset_index()
    
    x.columns = ['time'] + ['species_%d' % j for j in range(1, len(x.columns))]
    
    df_ts[keys[i]] = x


# David stool data

files = ['Data/Faust/25_timeseries/25_timeseries.txt']
keys = ['David_stool_A']

for i, f in enumerate(files):
    x = pd.read_csv(f, na_values='NAN', delimiter='\t', header=None)
    
    x = x.T
    
    x.insert(0, 'time', range(len(x)))
    
    x.columns = ['time'] + ['species_%d' % j for j in range(1, len(x.columns))]
    
    df_ts[keys[i]] = x
    
# Caporaso body sites data

sites = ['F4_L_palm_L6', 'F4_tongue_L6']

for site in sites:
    file = 'Data/Caporaso/' + site + '.txt'
    key = 'Caporaso_' + site

    x = pd.read_csv(file, delimiter='\t', skiprows=1, index_col=0, header=None)
    #x = x[x.sum(axis='rows') > 0]

    x.index = ['time'] + ['species_%d' % j for j in range(1, len(x.index))]

    x = x.T

    # only keep 200 most abundant species
    if len(x.columns) > 201:
        sum_x = x.sum(axis='rows')

        sum_x['time'] = np.inf

        sum_x.sort_values(ascending=False, inplace=True)

        x = x[sum_x.index.tolist()[:201]]

    x.columns = ['time'] + ['species_%d' % j for j in range(1, len(x.columns))]

    df_ts[key] = x

#calculate noise color for each exp timeseries
df_ns = {}

keys = ['plankton_eukarya', 'David_stool_A',
        'Caporaso_F4_L_palm_L6', 'Caporaso_F4_tongue_L6']

for i, key in enumerate(keys):
    ts = df_ts[key]
    df_ns[key] = noise_color(ts)['slope_linear']

#calculate width of distribution 

df_disdx = {}

#keys = ['Caporaso_F4_L_palm_L6'] 
keys =  ['plankton_eukarya', 'David_stool_A',
        'Caporaso_F4_L_palm_L6', 'Caporaso_F4_tongue_L6']

def fit_ratio(x):
    # ratios of succesive time points
    x = x = [x1/x2 for x1, x2 in zip(x[:-1], x[1:]) if x1 != 0 and x2 != 0 ] 
    
    if len(x) > 5:
        a, b, c = stats.lognorm.fit(x, floc=0)  # Gives the paramters of the fit
        stat, pval = stats.kstest(x, 'lognorm', args=((a, b, c))) # get pvalue for kolmogorov-smirnov test 
        # (null hypothesis: ratios of succesive time points follow lognorm distribution)

        return a, b, c, stat, pval
    else:
        return (np.nan, np.nan, np.nan, np.nan, np.nan)

count = 0

for i, key in enumerate(keys):
    ts = df_ts[key]

    dx_ratio = pd.DataFrame(index=ts.columns, columns=['s', 'loc', 'scale', 'ks-stat', 'ks-pval'])
    dx_ratio.drop('time', inplace=True)

    for idx in dx_ratio.index:
        fit_par = fit_ratio(ts[idx].values)
        dx_ratio.loc[idx] = fit_par
        
        if False and fit_par[-1] > 0.5 and count < 10:
            print(key, idx, fit_par[-1])
            
            print(x[:5])

            x = ts[idx].values
            x_transf = x[:-1] / x[1:] # ratios of succesive time points
            x_transf = x_transf[np.isfinite(x_transf)] # remove infinities
            
            a, b, c, _, pval = fit_par
            
            x_fit = np.logspace(-1.5,1.5,100)
            pdf_fitted = stats.lognorm.pdf(x_fit,a,b,c) #Gives the PDF
            plt.figure()
            plt.hist(x_transf, alpha=0.4, normed=True, bins = np.logspace(-1.5,1.5,30))
            plt.plot(x_fit, pdf_fitted, label='%.2f, %.2f, %.2f'%(a,b,c))
            plt.xscale('log')
            plt.legend()
            plt.show()
            
            count += 1
    
    if count == 10:
        break;
        
    df_disdx[key] = dx_ratio

plotts = True
plotra = True
plotnc = True
plotdx = True
plotdisdx = True

keys = ['plankton_eukarya', 'David_stool_A',
        'Caporaso_F4_L_palm_L6', 'Caporaso_F4_tongue_L6']

titles = ['Plankton eukarya', 'Microbiome stool', 
          'Microbiome palm', 'Microbiome tongue']

fig = plt.figure(figsize=(0.9*ELIFE.FULLWIDTH,7))

lm = 0.15 # left margin
rm = 0.85 # right margin

gs_ts = gridspec.GridSpec(2, len(keys), top=0.97, bottom=0.7, left = lm, right=rm, hspace=0.5, wspace=0.1)
gs_ma = gridspec.GridSpec(3, len(keys), top=0.65, bottom=0.22, left = lm, right=rm, hspace=0.05, wspace=0.1)
gs_legend = gridspec.GridSpec(2, 1, top=0.97, bottom=0.7, left = 0.88, right=0.99, hspace=0.5, wspace=0.1)
gs_cbar = gridspec.GridSpec(3, 1, top=0.65, bottom = 0.22, left = 0.88, right=0.9, hspace=0.05, wspace=0.1)

gs1 = gridspec.GridSpec(2,1,hspace=0, left=lm, right= rm, top=0.15, bottom=0.11) # for neutrality
gs2 = gridspec.GridSpec(1,2,wspace=0.2, left=lm, right = rm, top = 0.07, bottom=0.05) # for colorbars neutrality

#axes = np.empty([5, len(keys)])

axes = [[0 for i in range(len(keys))] for j in range(5)]
for i in range(2):
    for j in range(len(keys)):
        if j == 0: # share axes except for timeseries
            axes[i][j] = fig.add_subplot(gs_ts[i,j])
            if i == 0:
                axes[i][j].set_title(titles[i])
        elif i == 0:
            axes[i][j] = fig.add_subplot(gs_ts[i,j], sharey=axes[i][0])
            axes[i][j].set_title(titles[j])
        else:
            axes[i][j] = fig.add_subplot(gs_ts[i,j], sharey=axes[i][0], sharex=axes[i][0])
        axes[i][j].grid()
        
for i in range(2,5):
    for j in range(len(keys)):
        if i-2 == 0 and j == 0: # share axes except for timeseries
            axes[i][j] = fig.add_subplot(gs_ma[i-2,j])
        elif i-2 == 0:
            axes[i][j] = fig.add_subplot(gs_ma[i-2,j], sharey=axes[i][0], sharex=axes[i][0])
        elif j == 0:
            axes[i][j] = fig.add_subplot(gs_ma[i-2,j], sharex=axes[i-1][j])
        else:
            axes[i][j] = fig.add_subplot(gs_ma[i-2,j], sharey=axes[i][0], sharex=axes[i-1][j])
        axes[i][j].grid()
axes = np.array(axes)

axes_cbar = fig.add_subplot(gs_cbar[-1])
axes_legend = fig.add_subplot(gs_legend[-1])
axes_legend.axis('off')

for i, key in enumerate(keys):
    ts = df_ts[key]
    mean = df_ts[key].mean()
    mean.drop('time', inplace=True)
    ts['time'] -= ts['time'].min()
    
    vmin = 1e-4
    vmax = 1e5

    # timeseries
    
    if plotts:
        ax = axes[0,i]

        sorted_species = mean.sort_values().index.tolist()[::-1]

        skip = max(1, int(len(ts) / 500))
        for species in sorted_species[::int((len(ts.columns)-1) / 4)]:
            ax.plot(ts['time'][::skip], ts[species][::skip])
        
        ax.set_yscale('log')
        
    # Rank abundance
    
    if plotra:
        ax = axes[1][i]

        selected_times = np.arange(ts['time'].min(), ts['time'].max(), 50)[:4]
        
        for t in selected_times:
            abundance_profile = ts[ts['time'] == t-1].values.flatten()[1:]
            ax.plot(range(1, len(abundance_profile) + 1), np.sort(abundance_profile)[::-1],
                       label='Day %d' % int(t))
        

        ax.set_xscale('log')
        ax.set_yscale('log')
        if i == 1:
            handles, labels = ax.get_legend_handles_labels()
            axes_legend.legend(handles, labels, handlelength=1, fontsize=ELIFE.FONTSIZE)
        ax.set_ylim([vmin, vmax])
    
    # Noise color
    
    if plotnc:
        ax = axes[2][i]

        ax.set_xscale('log')

        ns = df_ns[key]
        sc = ax.scatter(mean, ns, vmin=0, vmax=10, s=3)

        xx = np.linspace(2, -3, 500).reshape([500, 1])
        ax.imshow(xx, cmap=noise_cmap_ww, vmin=noise_lim[0], vmax=noise_lim[1], extent=(vmin, vmax, -3, 2),
                     aspect='auto', alpha=0.75)
        ax.set_xlabel('Abundance')
        
        plt.setp(ax.get_xticklabels(), visible=False)
        
    # absolute timestep
    
    if plotdx:
        ax = axes[3][i]

        dx = (ts.values[1:, 1:] - ts.values[:-1, 1:])  # / x.values[:-1, 1:];
        dx[~np.isfinite(dx)] = np.nan
        mean_dx = np.nanmean(abs(dx), axis=0)

        p_lin = np.polyfit(np.log10(mean), np.log10(mean_dx), deg=1, cov=False)

        xx = [np.nanmin(mean.values), np.nanmax(mean.values)]
        ax.plot(xx, 10 ** (p_lin[1] + p_lin[0] * np.log10(xx)), c='k', linewidth=0.5)
        ax.annotate(r'y $\propto$ x$^{%.2f}$' % p_lin[0],(0.3,0.01))
        ax.scatter(mean, mean_dx, s=3)

        ax.set_xscale('log')
        ax.set_yscale('log')

        ax.set_xlabel('Mean abundance')
        plt.setp(ax.get_xticklabels(), visible=False)
        #ax.set_xticklabels(['']*10) # no ticklabels
    
    # distribution timestep
    
    if plotdisdx:
        ax = axes[4][i]
        
        dx_ratio = df_disdx[key]
        
        sc = ax.scatter(mean, dx_ratio['s'], c=dx_ratio['ks-pval'], vmin=0, vmax=1, cmap='coolwarm', s=3)

        # ax_disdx.legend()
        ax.set_xscale('log')
        ax.set_yscale('log')
        
        if i == 0:
            fig.colorbar(sc, cax=axes_cbar)
            axes_cbar.set_ylabel('p-value lognormal fit')
        
    if i == 0:
        axes[0][i].set_ylabel('Abundance')
        axes[0][i].set_ylim([1e-4,1e5])
        axes[1][i].set_ylabel('Abundance')
        axes[1][i].set_ylim([1e-1,1e3])
        axes[1][i].set_xlim([5e-1,3e2])
        axes[2][i].set_ylabel('Slope power \n spectral density')
        axes[3][i].set_ylabel('Difference \n time points \n' + r'$\left< \mid x(t+\delta t) - x(t) \mid \right>$')
        # \langle \rangle
            #'Mean absolute \n difference between successive \n time points')
        axes[3][i].set_ylim([1e-3, 5e4])
        axes[4][i].set_ylabel('Width distribution \n of ratios \n' + r'$x(t + \delta t) / x(t)$')
        axes[4][i].set_xlim([1e-3, 5e4])
        axes[4][i].set_ylim([8e-2, 8e0])
    else:
        for j in range(5):
            plt.setp(axes[j][i].get_yticklabels(), visible=False)
    
    if i == len(keys) - 1:
        axes[0][i].set_xlabel('Time (days)', ha='right', x=1)
        axes[1][i].set_xlabel('Rank', ha='right', x=1)
        axes[1][i].set_xticks([10,100])
        axes[1][i].set_xticklabels([10,100])
        axes[4][i].set_xlabel('Mean abundance', ha='right', x=1)
        axes[4][i].set_xticks([1e-2, 1e1, 1e4])

for ax, label in zip(axes[:,0], ('A', 'B', 'C', 'D', 'E')):
    ax.text(-0.72, 1.05, label, transform=ax.transAxes,
      fontsize=10, fontweight='bold', va='top', ha='right')
    
# neutrality

neutrality =pd.read_csv('results/experimental/neutrality.csv', index_col=0)
neutrality = neutrality.loc[keys]

ax_KL = fig.add_subplot(gs1[0])
ax_clb_KL = fig.add_subplot(gs2[0])
ax_NCT = fig.add_subplot(gs1[1])
ax_clb_NCT = fig.add_subplot(gs2[1])
ax_KL.set_facecolor('lightgrey')
ax_NCT.set_facecolor('lightgrey')

ax_KL.text(-0.72*0.23, 1.05, 'F', transform=ax_KL.transAxes,
      fontsize=10, fontweight='bold', va='top', ha='right')

x = np.log10(neutrality['KL'].values.astype(np.float64))
x = x.reshape([1, len(x)])
x[np.isinf(x)] = 3.0
mat_KL = ax_KL.matshow(x, origin='lower', 
                    cmap='Blues_r', aspect='auto', vmin=-1, vmax=3)
ax_KL.set_yticks([0])
ax_KL.set_yticklabels([r'log$_{10}$($D_{KL}$)'])

ax_KL.tick_params(axis="both", bottom=False, top=False, labelbottom=False, labeltop=False, left=True, labelleft=True)

fig.colorbar(mat_KL, cax=ax_clb_KL, orientation='horizontal')
#ax_clb_KL.set_title(r'log$_{10}$($D_{KL}$)')
ax_clb_KL.set_xlabel(r'log$_{10}$($D_{KL}$)', ha='right', x=1)

x = np.log10(neutrality['NCT'].values.astype(np.float64))
x = x.reshape([1, len(x)])

vmin = -5; vmax = 0 # pvalue is max 1 = 1e0
norm = PiecewiseNormalize([vmin, np.log10(0.05), vmax], [0, 0.5, 1])
mat_NCT = ax_NCT.matshow(x, origin='lower', norm=norm, 
                     cmap='seismic_r', aspect='auto', vmin=vmin, vmax=vmax)
fig.colorbar(mat_NCT, cax=ax_clb_NCT, orientation='horizontal')

ax_NCT.set_yticks([0])
ax_NCT.set_yticklabels([r'log$_{10}$($p_{NCT}$)'])

# Remove ticks
ax_NCT.tick_params(axis="both", bottom=False, top=False, labelbottom=False, labeltop=False, left=True, labelleft=True)

ax_clb_NCT.set_xlabel(r'log$_{10}$($p_{NCT}$)', ha='right', x=1)
ax_clb2 = ax_clb_KL.twiny()
ax_clb_KL.xaxis.set_ticks_position('bottom')
#ax_clb2.xaxis.set_ticks_position('top')
ax_clb2.xaxis.set_ticks([0.05,0.95])
ax_clb2.set_xlim([0,1])
ax_clb2.xaxis.set_ticklabels(['neutral','niche'])
ax_clb2.tick_params(axis='x', direction='out')

ax_clb2 = ax_clb_NCT.twiny()
ax_clb_NCT.xaxis.set_ticks_position('bottom')
ax_clb2.xaxis.set_ticks_position('top')
#ax_clb2.xaxis.set_tick_params(direction='out', which='top')
ax_clb2.xaxis.set_ticks([1+(vmin + np.log10(0.05))/(vmax - vmin)/2,
                        1+(vmax + np.log10(0.05))/(vmax - vmin)/2])
ax_clb2.set_xlim([0,1])
ax_clb2.xaxis.set_ticklabels(['niche','neutral'])
ax_clb2.tick_params(axis='x', direction='out')
    
#fig.align_labels()
fig.align_ylabels(axes[:,0])

plt.show()
### Characteristics of experimental data.

Box 1.

Definitions of the studied characteristics We study multiple characteristics of the dynamics of microbial communities.

We here define these characteristics. The labels (A-F) denote the different figures of Figure 1 and Figure 4.

A. A time series represents the time evolution of the abundances of different species of the community.

B. The rank abundance distribution describes the commonness and rarity of all species. It can be represented by a rank abundance plot, in which the abundances of the species are given as a function of the rank of the species, where the rank is determined by sorting the species from high to low abundance. These curves can generally be fitted with power law, lognormal, or logarithmic series functions 18Limpert et al., 200124McGill et al., 20071Brown et al., 2002.

C. The noise color describes the distribution of the frequencies of the fluctuations of a time series of a species. It is defined by the slope of a linear fit through the power spectral density. White, pink, brown and black noise correspond to slopes around 0,–1, −2 and −3 respectively. The more negative the slope is—this corresponds to darker noise—the more structure there is in the time series 7Faust et al., 2018.

D. We study the mean absolute difference between abundances at successive time points$⟨\mid x(t+\delta t)-x\left(t\right)\mid ⟩$ as a function of the mean abundance $⟨x\left(t\right)⟩$. These values represent the jumps of the abundances from one time point to the next.

E. We measure the ratios of the abundances at two successive time points $x(t+\delta t)/x\left(t\right)$. The advantage of this method is that it captures the direction of a jump between two time points: for ratios higher than one the jump is positive, for ratios lower than one the jump is negative. The distribution of these ratios fits a lognormal curve with a mean at one as the fluctuations occur around steady-state and the width of the distribution tells how large the fluctuations of a time series are. The goodness of the fit is defined by the p-value of the Kolmogorov-Smirnov test. Higher p-values denote a better fit. We use the width as a characteristic and compare the widths of different species. Examples of the fitted lognormal curve can be found in Supplementary file 1: Supporting results.

F. The Kullback-Leibler divergence measures how different the multivariate distribution of species abundances is from a distribution constructed under the assumption of ecological neutrality. The idea of the neutral covariance test is to compare the time series with a Wright-Fisher process. A Wright-Fisher process is a continuous approximation of Hubbell’s neutral model for a large and finite community. In particular, it tests the invariance with respect to grouping. More about the validity of these neutrality measures can be found in the Supplementary file 1: Supporting results.

The time series show fluctuations over time

The experimental time series show large fluctuations over time. We can ask the question whether the origin of this variation is biological or technical, and assume that most of the variation can be contributed to biological processes. This hypothesis is supported by the results of 29Silverman et al., 2018 for microbial communities of an artificial gut. Here, the biological variation becomes five to six times more important than the technical variation for the sampling interval of a day. Also, 15Grilli, 2019 shows the time correlation of experimental time series which is non-zero. In the case where the variation is mostly due technical errors, we expect to see no correlation. Because no experimental errorbars are available for most of the data and because we assume most variation has a biological origin, we did not consider the errors on the species abundances.

The abundance distribution is heavy-tailed

The first aspect of community modeling that has been widely studied during the last years is the stability of the steady-states. Large random networks tend to be unstable 22May, 1972. This problem is often solved by considering only weak interactions, sparse interaction matrices 23May, 2001 or by introducing higher-order interactions 14Grilli et al., 20179Gavina et al., 201828Sidhom and Galla, 2019. Although the stability of gLV models decreases with an increasing number of participating species, the stability only depends on the interaction matrix and not on the abundances 10Gibbs et al., 2018. The abundance distribution of the experimental data is heavy-tailed. This means that there are few common and many rare species. The distribution of the steady-state values can also be represented by a rank abundance curve (see Box 1B). Although the abundances show large fluctuations over time, the rank abundance remains stable (Figure 1B).

The differences between abundances at successive time points are large and linear with respect to the species abundance

Time series can be described by the differences between abundances at successive time points. We propose to focus on two specific representations of the information contained in those differences. First, we consider the mean absolute difference between abundances at successive time points $⟨\mid x(t+\delta t)-x\left(t\right)\mid ⟩$ as a function of the mean abundance $⟨x\left(t\right)⟩$ (see Box 1D). For the experimental data, the relation between these variables is a monomial—this means that it is linear on the log-log scale (Figure 1D). The fact that the slope of this line is almost one hints at a linear nature of the noise.

Second, we examine the distribution of the ratios of the abundances at two successive time points $x(t+\delta t)/x\left(t\right)$ (see Box 1E). The width of this distribution tells how large the fluctuations are. To measure this width, we fit the distribution with a lognormal curve for which the mean is fixed to be one as the fluctuations occur around steady-state. For most of the species of experimental data (except for the stool data), the fit of the distribution to a lognormal curve is good (Figure 1E). Furthermore, we notice that the distribution is wide—in the order of 1—and that the width does not depend on the mean abundance of the species (Figure 1E).

The noise color is independent of the mean abundance of the species

The noise of a time series can be studied by considering the distribution of the frequencies of the fluctuations. This distribution can be defined by its slope, which is interpreted as the noise color (see Box 1C). We notice that there is no correlation between the noise color and the mean abundance of the species for experimental time series (Figure 1C).

Experimental time series are in the niche regime

In neutral theory, it is assumed that all species or individuals are functionally equivalent. It is challenging to test whether a given time series was generated by neutral or niche dynamics. We use two definitions of neutrality measures: the Kullback-Leibler divergence as used in 8Fisher and Mehta, 2014 and the neutral covariance test as proposed by 33Washburne et al., 2016 (see Box 1F). Both neutrality measures indicate that most experimental time series are in the niche regime (Figure 1F).

Reproducing properties of experimental time series from stochastic generalized Lotka-Volterra models

We find that the aforementioned characteristics of experimental time series can be reproduced by stochastic logistic equations. We first explain how to choose the growth rate to obtain the heavy-tailed experimental abundance distribution. Next, we discuss how the noise color determines the self-interaction of a species given its abundance and how the implementation of the noise determines the slope of the mean absolute increment $⟨\mid x(t+\delta t)-x\left(t\right)\mid ⟩$ and the mean abundance $⟨x\left(t\right)⟩$ (such as in Figure 1D). In the end, by using the appropriate choice for the self-interactions, growth rates, and noise implementation, we conclude that a stochastic logistic model can reproduce all the stochastic properties, including the niche regime for the neutrality tests although the model does not include any interactions.

The rank abundance distribution can be imposed by fixing the growth rate

Random matrix models do typically not give rise to heavy-tailed abundance distributions. Neither is it known which properties of the interaction matrix and growth rates are required to obtain a realistic rank abundance distribution. We can however enforce the desired rank abundance artificially by solving the steady-state of the gLV equations. Given the steady-state abundance vector ${\overrightarrow{x}}^{\ast }$ and interaction matrix ω, we impose the growth rate $\hat{g}=-\omega {\overrightarrow{x}}^{\ast }$. One model that results in heavy-tailed distributions is the self-organized instability model proposed by 30Solé et al., 2002.

For logistic models, the growth rate is equal to the product of the self-interaction and mean abundance. The noise color and the width of the distribution of ratios $x(t+\delta t)/x\left(t\right)$ depend on this product. To obtain given characteristics—a predefined noise color and width of the distribution of ratios $x(t+\delta t)/x\left(t\right)$—the choice of the growth rate will dictate the choice of the remaining free parameters, the sampling time step δt and the noise strength σ.

The noise color is determined by the mean abundance and the self-interaction of the species

To study the noise color, we first consider a model where the species are not interacting. The noise color is independent of the implementation of the noise but depends on the product of the mean abundance and the self-interaction of the species (Figure 2A). For noninteracting species, the growth rate equals the product of the self-interaction and the steady-state abundance. Because we consider fluctuations around steady-state, the mean and the steady-state abundance are nearly equal and the x-axis of Figure 2A; Figure 2B; Figure 2C; can be interpreted as the growth rate. Also, the strength of the noise does not change its color (Figure 2C). A parameter that is important for the noise color is the sampling rate: the higher the sampling frequency the darker the noise becomes (Figure 2B). This is in agreement with the results of 7Faust et al., 2018. Darker noise corresponds to more structure in the time series. The more frequent the abundances are sampled the more details are visible and the underlying interactions become more visible. We conclude that the noise color is only dependent on the mean abundance, the self-interactions, and the sampling rate. Figures of the dependence on the mean abundance and self-interaction separately can be found in Supplementary file 1: Supporting results.

Figure 2

fig = plt.figure(figsize=(ELIFE.TEXTWIDTH, 4))

ymin = -3
ymax = 1.5

# without interactions

yoff = 0.01

gs = gridspec.GridSpec(1, 3, top=0.88+yoff, bottom=0.6+yoff,
                       left=0.1, right=0.95, wspace=0.05, hspace=0.05)

gs_comb = gridspec.GridSpec(
    1, 1, top=0.95+yoff, bottom=0.55+yoff, left=0.07, right=0.95)

ax = fig.add_subplot(gs_comb[0], frameon=False)
ax.set_xlabel(r'Mean abundance $\times$ self-interaction', ha='right', x=1)
ax.set_ylabel('Slope power \n spectral density')
ax.set_xticks([])
ax.set_yticks([])

# ax.text(-0.02, 1.05, 'A', transform=ax.transAxes,
#      fontsize=10, fontweight='bold', va='top', ha='right')
ax.text(0, 1.08, 'Logistic model (without interactions)', transform=ax.transAxes,
        fontsize=9, va='top', ha='left')
# ax.text(0.5, 1.1, 'Logistic model (without interactions)', transform=ax.transAxes,
#        fontsize=9, va='top', ha='center')

# implementation

ax = fig.add_subplot(gs[0])
ax.text(0, 1.05, 'A', transform=ax.transAxes,
        fontsize=10, fontweight='bold', va='bottom', ha='left')

path = 'results/noise_color/no_interaction/'
files_noise = [path + 'noise_abundance_Langevin_linear.csv',
               path + 'noise_abundance_Langevin_sqrt.csv',
               path + 'noise_abundance_Langevin_constant.csv'][::-1]
# [path + 'noise_abundance_Ricker_linear.csv',
#[path + 'noise_abundance_Arato_linear.csv'][::-1]

# , 'Ricker linear', 'Arato linear'][::-1]
labels = ['Linear multiplicative', 'Sqrt multiplicative', 'Additive']

PlotNoiseColorComparison(
    files_noise, labels, legend_title='Noise implementation', ax=ax, masi=True)
ax.set_xlabel('')
ax.set_ylabel('')
ax.set_ylim([ymin, ymax])
ax.set_xlim([2e-2, 2e2])

# sampling dt

ax = fig.add_subplot(gs[1], sharex=ax, sharey=ax)
ax.text(0, 1.05, 'B', transform=ax.transAxes,
        fontsize=10, fontweight='bold', va='bottom', ha='left')

files_noise_samp = [
    path + 'noise_abundance_Langevin_samp%d.csv' % i for i in range(1, 5)]

labels = ['0.005', '0.01', '0.025', '0.05']  # , '0.25']

PlotNoiseColorComparison(
    files_noise_samp[::-1], labels[::-1], legend_title='Sampling dt', ax=ax, masi=True)
ax.set_xlabel('')
ax.set_ylabel('')
ax.set_ylim([ymin, ymax])
ax.tick_params(axis="both", left=True, labelleft=False)
ax.set_xlim([2e-2, 2e2])

# noise strength

ax = fig.add_subplot(gs[2], sharex=ax, sharey=ax)
ax.text(0, 1.05, 'C', transform=ax.transAxes,
        fontsize=10, fontweight='bold', va='bottom', ha='left')

files_noise_sigma = [
    path + 'noise_abundance_Langevin_linear_sigma%d.csv' % i for i in range(1, 6)]

labels = ['0.01', '0.1', '0.2', '0.25', '0.3']

PlotNoiseColorComparison(
    files_noise_sigma[::-1], labels, legend_title='Noise strength $\sigma$', ax=ax, masi=True)
ax.tick_params(axis="both", left=True, labelleft=False)
ax.set_xlabel('')
ax.set_ylabel('')

# with interactions

gs = gridspec.GridSpec(1, 2, top=0.38+yoff, bottom=0.08 +
                       yoff, left=0.1, right=0.95, wspace=0.05, hspace=0.05)

gs_comb = gridspec.GridSpec(
    1, 1, top=0.45+yoff, bottom=0.03+yoff, left=0.07, right=0.95)

ax = fig.add_subplot(gs_comb[0], frameon=False)
ax.set_xlabel(r'Mean abundance $\times$ self-interaction', ha='right', x=1)
ax.set_ylabel('Slope power \n spectral density')
ax.set_xticks([])
ax.set_yticks([])

# ax.text(-0.02, 1.05, 'B', transform=ax.transAxes,
#      fontsize=10, fontweight='bold', va='top', ha='right')
ax.text(0, 1.08, 'Generalized Lotka-Volterra model (with interactions)', transform=ax.transAxes,
        fontsize=9, va='top', ha='left')
# ax.text(0.5, 1.05, 'Generalized Lotka-Volterra model (with interactions)', transform=ax.transAxes,
#        fontsize=9, va='top', ha='center')

norm = mpl.colors.Normalize(vmin=0, vmax=0.21, clip=True)
mapper = mpl.cm.ScalarMappable(norm=norm, cmap='summer')

ax = fig.add_subplot(gs[0])
ax.text(0, 1.05, 'D', transform=ax.transAxes,
        fontsize=10, fontweight='bold', va='bottom', ha='left')
ax.text(0.1, 1.05, 'Equal abundances', transform=ax.transAxes,
        fontsize=ELIFE.FONTSIZE, va='bottom', ha='left')
#ax.set_title('Equal abundances')

path = 'results/noise_color/with_interaction/'
files_noise_int = [
    path + 'noisecolor_Langevin_linear_interaction%d.csv' % i for i in [1, 2, 3, 6]]

labels = ['0.01', '0.05', '0.1', '0.15']

PlotNoiseColorComparison(files_noise_int, labels,
                         legend_title=r'Interaction strength $\alpha$', ax=ax, masi=True, interaction_colors=True)

ax.set_ylabel('')
ax.set_xlabel('')
ax.set_ylim([ymin, ymax])

ax = fig.add_subplot(gs[1], sharex=ax, sharey=ax)
ax.text(0, 1.05, 'E', transform=ax.transAxes,
        fontsize=10, fontweight='bold', va='bottom', ha='left')
ax.text(0.1, 1.05, 'Lognormally distributed abundances', transform=ax.transAxes,
        fontsize=ELIFE.FONTSIZE, va='bottom', ha='left')
#ax.set_title('Lognormally distributed abundances')

files_noise_pl = [path + 'noisecolor_Langevin_linear_powerlaw_sigma1.csv',
                  path + 'noisecolor_Langevin_linear_powerlaw_sigma2.csv',
                  path + 'noisecolor_Langevin_linear_powerlaw_sigma3.csv',
                  path + 'noisecolor_Langevin_linear_powerlaw_sigma4.csv']

labels = ['0', '0.1', '0.15', '0.2']

PlotNoiseColorComparison(files_noise_pl, labels,
                         legend_title=r'Interaction strength $\alpha$', ax=ax, masi=True, interaction_colors=True)
ax.tick_params(axis="both", left=True, labelleft=False)
ax.set_ylabel('')
ax.set_xlabel('')
ax.set_ylim([ymin, ymax])

plt.show()

findfont: Font family ['Open Sans'] not found. Falling back to DejaVu Sans.

For interacting species, increasing the strength of the interactions makes the color of the noise darker in the high mean abundance range (Figure 2D; Figure 2E). Importantly, for interacting species with a lognormal rank abundance, the correlation between the noise color and mean abundance is preserved (Figure 2E). The data can be fit to obtain a bijective function between the product of the mean abundance and the self-interaction, and the noise color. Assuming this model is correct, we can obtain an estimate for the self-interaction coefficients given the mean abundance and noise color by fixing the sampling rate and the interaction strength. The uncertainty on the estimates is larger where the fitted curve is more flat (slopes of the power spectral density around −1.7 and 0), but many experimental values of the stool microbiome data lie in the pink region where the self-interaction can be estimated for this model.

The implementation of the noise determines the correlation between the mean absolute increment $⟨\mid x(t+\delta t)-x\left(t\right)\mid ⟩$ and the mean abundance $⟨x\left(t\right)⟩$

Next, we study the differences between abundances at successive time points (see Figure 1D). From the results of the noise color, we can estimate the self-interaction for the dynamics of the experimental data. We use the rank abundance and the self-interaction inferred from noise color of the microbiome data of the human stool to perform simulations and calculate the characteristics of the distribution of differences between abundances at successive time points. We here assume that there are no interactions. More results for dynamics with interactions are in Supplementary file 1: Supporting results. We first study the correlation between the mean absolute difference between abundances at successive time points $⟨\mid x(t+\delta t)-x\left(t\right)\mid ⟩$ and the mean abundance $⟨x\left(t\right)⟩$. For linear multiplicative noise, the slope of the curve of the logarithm of the mean absolute difference between abundances at successive time points ${log}_{10}(⟨\mid x(t+\delta t)-x(t)\mid ⟩)$ as a function of the logarithm of the mean abundance ${log}_{10}(⟨x(t)⟩)$ is one. For multiplicative noise that scales with the square root of the abundance, the slope is around 0.66 and for additive noise, the slope is zero. By combining both linear noise and noise that scales with the square root of the abundance, slopes with values between 0.6 and 1 can be obtained (Figure 3A). The slopes of experimental data range between 0.84 and 0.99, we therefore conclude that linear noise is a relatively good approximation to perform stochastic modeling of microbial communities.

Figure 3

path = 'results/width_ratios/'
df1 = pd.read_csv(path +  'width_lognormal_fit_1.csv')
df2 = pd.read_csv(path +  'width_lognormal_fit_1_interaction0.05.csv')
df3 = pd.read_csv(path +  'width_lognormal_fit_1_interaction0.1.csv')
df4 = pd.read_csv(path +  'width_lognormal_fit_1_interaction0.15.csv')

sigmas = [0.01, 0.1, 1.0]

cmap = mpl.cm.get_cmap('coolwarm') #viridis')

norm = mpl.colors.Normalize(vmin=0, vmax=0.21, clip=True)
mapper = mpl.cm.ScalarMappable(norm=norm, cmap='summer')

fig = plt.figure(figsize=(ELIFE.TEXTWIDTH,2.3))

gs_l = gridspec.GridSpec(2,1, height_ratios=[8,1], hspace= 0.8, 
                         right=0.95, left=0.8, top=0.85, bottom=0.2)
gs_r = gridspec.GridSpec(2,2, height_ratios=[8,1], hspace= 0.8, wspace=0.05, 
                         right=0.65, left=0.15, top=0.85, bottom=0.2)

ax_mat = fig.add_subplot(gs_l[0])
ax = fig.add_subplot(gs_r[0])
ax2 = fig.add_subplot(gs_r[1], sharey=ax)

ax_mat.text(-0.2, 1.08, 'B', transform=ax_mat.transAxes,
      fontsize=10, fontweight='bold', va='bottom', ha='right')
ax.text(-0.2, 1.08, 'A', transform=ax.transAxes,
      fontsize=10, fontweight='bold', va='bottom', ha='right')

ax_mat_cbar = fig.add_subplot(gs_l[1])
ax_legend = fig.add_subplot(gs_r[2])
ax_cbar = fig.add_subplot(gs_r[3])


df_slopes2 = pd.read_csv('results/slopes/slopes_equal_abundances.csv', index_col=0, na_values='NAN')
df_slopes2['slope'] = df_slopes2.iloc[:,2:12].mean(axis=1)
df_slopes2['slope_std'] = df_slopes2.iloc[:,2:12].std(axis=1)
df_slopes2.drop(['%d'%i for i in range(10)], axis=1, inplace=True)

slope = df_slopes2.drop(['implementation', 'interaction', 'slope_std'], axis=1)
std_slope = df_slopes2.drop(['implementation', 'interaction', 'slope'], axis=1)

slope = slope.groupby(['noise_lin', 'noise_sqrt']).agg('mean')
std_slope = std_slope.groupby(['noise_lin', 'noise_sqrt']).agg('mean')

slope = slope.unstack() #.iloc[:4, :4]

val = slope.values

mat = ax_mat.matshow(val, cmap='coolwarm', vmin=0.65, vmax=1.1)
ax_mat.set_xlabel(r'$\sigma_\mathregular{sqrt}$')
ax_mat.set_ylabel(r'$\sigma_\mathregular{lin}$')
ax_mat.set_xticks([0,1,2,3,4])
ax_mat.set_yticks([0,1,2,3,4])

ax_mat.set_xticklabels([0, 0.01, 0.1, 0.5, 1.0], rotation=90)
ax_mat.set_yticklabels([0, 0.01, 0.1, 0.5, 1.0])

cbar = plt.colorbar(mat, cax=ax_mat_cbar, orientation='horizontal')
cbar.set_label(r'Slope $\left< \mid x(t+\delta t) - x(t) \mid \right>$') #'Slope steps')

#for i, df, alpha in zip(range(4), [df1, df2, df3, df4], [0, 0.05, 0.1, 0.15]):
for i, df, alpha in zip(range(3), [df1, df3, df4], [0, 0.1, 0.15]):
    for j, sigma in enumerate(sigmas):
        w = df['sigma_%.2f_width_mean' % sigma]
        pval = df['sigma_%.2f_pval' % sigma]
        ss = df['ss']
        
        col = mapper.to_rgba(alpha)
        
        ax.plot(ss.values, w.values, c=col, alpha=0.3, marker='o', markersize=3, label=alpha if j==0 else "")
        ax2.plot(ss.values, w.values, c='lightgrey', alpha=0.3) #, label=alpha if j==0 else "")
        s_ax2 = ax2.scatter(ss.values, w.values,s=3, c = pval, cmap=cmap, vmin=0, vmax=1)
                        #c=col, label=alpha if j==0 else "")
        x = 2e-1 #ss.values[0]
        y = w.values[0]
        
        if i == 0:
            ax.annotate(r"$\sigma_\mathregular{lin} =$ %.2f" % sigma, xy=(x, y), xytext=(0.2*x, 1.5*y))
            ax2.annotate(r"$\sigma_\mathregular{lin} =$ %.2f" % sigma, xy=(x, y), xytext=(0.2*x, 1.5*y))

handles, labels = ax.get_legend_handles_labels()
ax_legend.legend(handles, labels, title='Interaction ' + r'strength $\alpha$', 
                 loc=9, ncol=3, columnspacing=0.5)
ax_legend.axis('off')

cbar = plt.colorbar(s_ax2, cax=ax_cbar, orientation='horizontal')
cbar.set_label('p-value lognormal fit')

ax.set_xscale('log')
#ax.set_xlabel(r'Mean abundace $\times$ self-interaction', ha='right', x=1)

ax.set_ylabel('Width distribution \n of ratios \n' + r'$x(t + \delta t) / x(t)$') #'Width distribution \n ratios of time points')
ax.set_xlim([2e-2,2e2])
ax2.set_ylim([5e-4,1e0])
ax.set_yscale('log')
ax.grid()

ax2.set_xscale('log')
ax2.tick_params(axis="both", left=True, labelleft=False)
ax2.set_xlabel(r'Mean abundace $\times$ self-interaction', ha='right', x=1)
#ax2.set_ylabel('Scale lognormal fit')
ax2.set_xlim([2e-2,2e2])

ax2.set_yscale('log')
ax2.grid()

plt.show()

The strength of the noise determines the width of the distribution of ratios $x(t+\delta t)/x\left(t\right)$

Next, we examine the distribution of the ratios of abundances at successive time points (see Box 1E). As expected, for significant noise, this distribution can be approximated by a lognormal curve and the width of the distribution becomes larger for increasing noise strength (Figure 3B). In order to have widths that are of the same order of magnitude as the ones of the experimental data, the noise must be sufficiently strong. Another way of increasing the width is through interactions, this effect is only moderate. These results are presented in Supplementary file 1: Supporting results.

Stochastic logistic models capture the properties of experimental time series

By using all previous results and imposing the steady-state of experimental data, we find that it is possible to generate time series with identical characteristics to the ones seen in the experimental time series (Figure 4). Furthermore, these time series can be generated without introducing any interaction between the different species, but their neutrality measures can still be in the niche regime (Figure 4F). Out of 100 simulations, 62 had a p-value smaller than 0.05 for the neutral covariance test which means they are in the niche regime. The colors of the noise fix the self-interaction values (Figure 4C), next the rank abundance distribution is imposed by calculating the growth vector $\hat{g}$ (Figure 4B). The slope of the curve of the mean absolute difference between abundances at successive time points as a function of the mean abundance is one by using linear multiplicative noise (Figure 4D) and the width of the fluctuations is tuned by choosing a large noise size σ (Figure 4E). In most experimental time series, only the fractional abundances of species can be measured per time point and not the absolute ones. Because the total abundance of all species remains nearly constant in time series generated by a stochastic logistic equation, our results still hold for time series with fractional abundances (see Supporting results). Similar results can be obtained for models with interactions (see Supporting results), but we want to stress that interactions are not needed to reproduce the properties of experimental time series.

Figure 4

def mimic_experimental(interaction=0, connectivity=1, N=80):
    x = df_ts['David_stool_A'].values[150:, :]  # do not consider the traveling
    experimental_abundance = np.sort(x[0, :])[::-1]
    experimental_noise_color = noise_color(x.T)

    def find_ss_selfint(x):
        amplitude = 2.10E+00
        x0 = 2.87E+00
        k = 1.14E+00
        offset = -1.77E+00

        return 10**(-1/x0 * np.log(amplitude/(x-offset) - 1) + k)

    params = {}

    steadystate = (experimental_abundance[:N]).reshape([N, 1])

    selfints = - \
        find_ss_selfint(
            experimental_noise_color['slope_linear'].values[:N]) / steadystate.flatten()

    # interaction
    if interaction == 0:
        omega = np.zeros([N, N])
    else:
        omega = np.random.normal(0, interaction, [N, N])
        omega *= np.random.choice([0, 1], [N, N],
                                  p=[1-connectivity, connectivity])
    np.fill_diagonal(omega, selfints)

    params['interaction_matrix'] = omega

    # no immigration
    params['immigration_rate'] = np.zeros([N, 1])

    # different growthrates determined by the steady state
    params['growth_rate'] = - (omega).dot(steadystate)

    params['initial_condition'] = np.copy(
        steadystate) * np.random.normal(1, 0.1, steadystate.shape)

    params['noise'] = 2.5

    params['noise_linear'] = 2.5
    params['noise_sqrt'] = 0  # 0.005*steadystate #*np.sqrt(steadystate)
    
    np.save('test-params2.npy', params)
    
    ts = Timeseries(params, noise_implementation=NOISE.LANGEVIN_LINEAR_SQRT,
                    dt=0.01, tskip=19, T=50.0, seed=int(time.time())).timeseries
    ts.time = np.arange(1, len(ts)+1)

    return ts


def figure_characteristics_timeseries(ts):
    fig = plt.figure(figsize=(ELIFE.TEXTWIDTH, 3))

    gs1 = gridspec.GridSpec(1, 3, width_ratios=[
                            2.5, 2.5, 1], wspace=0.5, hspace=0.3, left=0.1, right=0.95, top=0.95, bottom=0.62)
    gs2 = gridspec.GridSpec(1, 3, wspace=0.7, hspace=0.4,
                            left=0.1, right=0.95, top=0.45, bottom=0.12)

    # timeseries
    ax = fig.add_subplot(gs1[0])
    ax.text(-0.2, 1.1, 'A', transform=ax.transAxes, fontsize=10,
            fontweight='bold', va='top', ha='right')
    ax.grid()

    PlotTimeseriesComparison([ts], composition=['ts'], vertical=False, fig=ax)

    ax = fig.add_subplot(gs1[1])
    ax.text(-0.2, 1.1, 'B', transform=ax.transAxes, fontsize=10,
            fontweight='bold', va='top', ha='right')
    ax.grid()
    # , ffig = 'figures/interaction_rescaled_model.png')
    PlotTimeseriesComparison([ts], composition=['ra'], fig=ax)
    ax.set_ylim([1e-2, 1e5])

    ax = fig.add_subplot(gs1[-1], frameon=False)
    ax.tick_params(left=False, labelleft=False,
                   bottom=False, labelbottom=False)
    ax.text(-0.5, 1.1, 'C', transform=ax.transAxes, fontsize=10,
            fontweight='bold', va='top', ha='right')

    sub_gs = gs1[0, -1].subgridspec(4, 1,
                                    height_ratios=[1.5, 1, 1, 1.5], hspace=0.3)
    ax_KL = fig.add_subplot(sub_gs[1])
    ax_NCT = fig.add_subplot(sub_gs[2])
    # , ffig = 'figures/interaction_rescaled_model.png')
    PlotTimeseriesComparison([ts], composition=['nn'], fig=[ax_KL, ax_NCT])

    # characteristics

    for i, (char, letter) in enumerate(zip(['nc', 'dx', 'disdx'], ['D', 'E', 'F'])):
        ax = fig.add_subplot(gs2[i])
        ax.text(-0.3, 1.1, letter, transform=ax.transAxes,
                fontsize=10, fontweight='bold', va='top', ha='right')

        ax.grid()
        # , ffig = 'figures/interaction_rescaled_model.png')
        PlotTimeseriesComparison([ts], composition=[char], fig=ax)
        if char == 'disdx':
            ax.set_ylim([1e-2, 1e2])
            ax.set_ylabel('Width distribution \n of ratios \n' +
                          r'$x(t + \delta t) / x(t)$')
        elif char == 'dx':
            ax.set_ylabel('Difference \n time points \n' +
                          r'$\left< \mid x(t+\delta t) - x(t) \mid \right>$')

    # fig.align_labels()

#KL = np.zeros(100)
#NCT = np.zeros(100)

if True:
    ts = mimic_experimental(interaction=0)
    figure_characteristics_timeseries(ts)
    plt.show()

findfont: Font family ['Open Sans'] not found. Falling back to DejaVu Sans.

Discussion

Recent research has focused on different aspects of experimental time series of microbial dynamics, in particular the rank abundance distribution, the noise color, the stability, and neutrality. Within the framework of stochastic generalized Lotka-Volterra models, we studied the influence of growth rates, interactions between species, and the different sources of stochasticity on the observed characteristics of the noise and on neutrality. Our observations are:

• Even when we consider the case without interactions between species, the result of the neutrality test on the time series is often niche. We should, therefore, be careful in the interpretation of the results of neutrality tests.
• For a given sampling step δt, the noise color depends on the product of the self-interaction and the mean abundance, which for noninteracting species reduces to a dependence on the growth rate. Assuming the model can be used for microbial communities, the self-interaction coefficients can be estimated given the mean abundance, noise color, and sampling rate. Low sampling rates result in larger errors (Figure 2B). For sparsely sampled experimental data, the standard deviation of the self-interaction inferred using the noise color will be larger. For the experimental time series (plankton, gut, and human microbiome) the self-interaction strengths range over several orders of magnitude. The convention of equalling all self-interactions to −1 used in several studies 8Fisher and Mehta, 201411Gibson et al., 2016, cannot be adopted for stochastic models of communities with a heavy-tailed abundance distribution.
• The exponent of the mean absolute differences between abundances at successive time points with respect to the mean abundances is slightly smaller than one for experimental time series. Linear multiplicative noise results in a value of one, square root noise results in lower values (0.6). A mix of linear and square root noise can result in slopes with intermediate values.
• A large multiplicative linear noise is in agreement with both the distribution of the ratios of abundances at successive time points and the relation between the differences between abundances at successive time points and mean values.

To conclude, characteristics of experimental time series, from plankton to gut microbiota, can be reproduced by stochastic logistic models with a dominant linear noise. We expect, however, that for higher sampling rates, modeling the interactions between microbes would be necessary to explain the properties of the time series. For gut microbial time series, the system is sampled only once a day and therefore dominated by the noise in the growth terms corresponding to a linear noise.

Predictive models for the dynamics of microbial communities will certainly require a more in-depth description of the system. Nutrients and spatial distribution of microbes should play a role to dictate the evolution of the community, as well as the interaction with the environment. Synthetic microbial communities are currently being developed and will hopefully provide a more comprehensive view on the complexity of microbial communities 31Vrancken et al., 2019.

Materials and methods

Modeling generalized Lotka-Volterra equations

In a microbial community different species interact because they compete for the same resources. Moreover, they produce byproducts that can affect the growth of other species. Depending on the nature of the byproducts, harmful, beneficial, or even essential, the interaction strength will be either negative or positive. To describe the dynamics of interacting species, one can use the generalized Lotka-Volterra equations:

where x_i, λ_i and g_i are the abundance, the immigration rate, and the growth rate of species i respectively, and ${\omega }_{ij}$ is the interaction coefficient that represents the effect of species $j$ on species i. The diagonal elements of the interaction matrix ${\omega }_{ii}$, the so-called self-interactions, are negative to ensure stable steady-states. The off-diagonal elements of the interaction matrix ${\omega }_{ij}$ are drawn from a normal distribution with standard deviation α (${\omega }_{ij}\sim N(0,{\alpha }^2)$). The gLV equations only consider pairwise effects and no saturation terms, or other higher-order terms. Due to this drawback, these models sometimes fail to predict microbial dynamics 25Momeni et al., 201717Levine et al., 2017. However, they are among the most simple models for interacting species and therefore widely studied and used. Noninteracting species can be described by the logistic model, which is a special case of the gLV model obtained by setting all off-diagonal elements of the interaction matrix to zero.

Implementations of the noise

There exist two principal types of noise: intrinsic and extrinsic noise. Extrinsic noise arises due to external sources that can alter the values of the different variables: the immigration rate and growth rate fluctuate in time through colonization of species or a changing flux of nutrients. These processes give rise to additive and linear multiplicative noise respectively. The remaining parameters, inter- and intra-specific interactions can also, change depending on the environment. The formulation of this noise is more subtle (used in 34Zhu and Yin, 2009). Intrinsic noise is due to the discrete nature of individual microbial cells. Thermal fluctuations at the molecular level determine the fitness of the individual cells. Therefore, cell growth, cell division, and cell death can be considered as stochastic Poisson processes. For large numbers of microbes, these fluctuations will be averaged out.

We first consider the extrinsic noise. If the time series is calculated by $x_i(t+dt)=x_i\left(t\right)+d{x}_i\left(t\right)$, the implementation of the linear multiplicative noise is as follows,

where dW is an infinitesimal element of a Brownian motion defined by a variance of dt ($dW\sim \sqrt{dt}N(0,1)$). Changes in immigration rates of microbial species can be modeled with additive noise,

with $d{W}_{\text{const}}\sim \sqrt{dt}N(0,1)$. Our main motivation is to model the gut microbiome in the colon. Here, we ignore the immigration of species for two reasons. First, the number of microbes in the colon is orders of magnitude larger than the number of microbes in the other parts of the gut 19Marteau et al., 200113Gorbach et al., 1967—therefore, the flux of incoming microbes in the colon is small. Second, we only consider systems around steady-state, for which we assume immigration does not play an important role. For perturbed systems, which are far from equilibrium, immigration rates cannot be ignored. Ignoring immigration may be too restrictive for some microbial systems such as the skin microbiome or plankton.

To derive the form of intrinsic noise in generalized Lotka-Volterra equations, we can consider every species abundance making a random walk in one dimension. The average displacement is zero and the variance of displacement is the sum of the rate of growth (jumping to the right) and the rate of death (jumping to the left). For the generalized Lotka-Volterra equations, this results in a noise term

with ω the interaction matrix and where functions f and h each decouple the growth and death terms. In the generalized Lotka-Volterra model no difference is made between negative interactions as a result of slowing down the growth rate or increasing the death rate, only the resulting net rates are used. This distinction must however be made to implement the intrinsic noise for gLV. In our analysis, we use the simpler logistic models where the resulting variance of the noise is proportional to the square root of the abundance $\sqrt{x}$. One must be careful not to use this noise for values that are smaller than one because this derivation relies on Poisson statistics which is defined for integer numbers.

We implement the intrinsic noise by a term that scales with the square root of the species abundance 32Walczak et al., 20128Fisher and Mehta, 2014,

with $d{W}_{\text{sqrt}}$ again an infinitesimal element of a Brownian motion defined by a variance of dt ($d{W}_{\text{sqrt}}\sim \sqrt{dt}N(0,1)$). The size of this noise ${\sigma }_{i,\text{sqrt}}$ is determined by the cell division (g⁺) and death rates (g^-) separately, which are in our model combined to one growth vector ($g=g^{+}-g^{-}$, ${\sigma }_{i,\text{sqrt}}=\sqrt{g^{+}+g^{-}}$), for large division and death rates the intrinsic noise will be larger.

To sum up, we focus on linear multiplicative noise because: (a) extrinsic noise is dominant as microbial communities contain a very large number of individuals and (b) we ignore the immigration of individuals in our analysis.

We verified that our analysis is robust with respect to the multiple possibilities for the discretization of these models. We also compare our population-level approach with individual-based modeling approaches. Details can be found in the Supplementary file 1: Supporting results.

Neutrality measures

There is no consensus on the definition of neutrality. In general, ecosystems are considered neutral if the dominating cause of fluctuations is random birth and death processes and not fitness advantages of species.

Different neutrality measures focus on different aspects of neutrality. The Kullback-Leibler divergence verifies whether all species are equal (equal abundances and equal covariances). The neutrality covariance test studies the grouping invariance of species in time series.

Given two distributions P and Q, the Kullback-Leibler divergence is defined as

where $E_P$ is the expectation value using the probabilities of distribution P. The density function of a multivariate Gaussian distribution is

where μ and K are the mean and covariance matrix of the distribution respectively. The Kullback-Leibler divergence for two multivariate Gaussian distributions in $R^n$ is 6Duchi, 2007

For every time series, we can calculate the mean μ and covariance matrix K, and define values ${\mu }_N$ and $K_N$ for a corresponding neutral time series in which all species are equal 8Fisher and Mehta, 2014. The distance to neutrality $D_{KL}\left(P|P_N\right)$ can thus be calculated by computing the probability distribution of the original time series P and the associated neutral distribution $P_N$ with mean values ${\mu }_N=S^{-1}{\sum }_{i=1}^S{\mu }_i$ and $K_{P,ii}=S^{-1}{\sum }_{i=1}^S{K}_{ii}$ and $K_{P,ij}=S^{-1}{(S-1)}^{-1}{\sum }_{i=1}^S{\sum }_{j=1,i\ne j}^S{K}_{ij}$ with S the number of species.

The neutral covariance test was designed by 33Washburne et al., 2016. We used a python translation of the code developed by this author.

Noise color

The color of the noise in a time series is determined by the slope of the power spectral density in a log-log scale. This slope can be determined by a linear fit through the spectrum. A different technique to estimate this slope has been put forward by 7Faust et al., 2018. There, it is argued that the power spectral density does not have a constant slope and that, therefore, a nonlinear curve must be fitted. They choose a spline fit and consider the minimal value of its derivative to be the value of the noise color. Because the minimal value of the slope of the fit is taken, the noise color tends to be darker when using this technique. For our time series, however, we see that the spline fit only deviates from the linear fit for low frequencies (Figure 5). We ignore the low frequencies for fitting because of the windowing effect. Therefore, we opt for a linear fit after omitting the values for low frequencies (one order of magnitude of the lowest frequencies).

ts = mimic_experimental(interaction=0.02, connectivity=0.1, N=50)
figure_characteristics_timeseries(ts)
plt.show()

The correspondence between the colors and slopes is here: