Adding %age memory and swap used to nad

6000110162

2015-08-21T13:01:07-04:00

6000171359

false

Data Collection 6

2015-10-30T11:00:31-04:00

2015-09-24T16:04:02-04:00

On a Linux box - If you would like to see a metric with percentage memory and swap used, it's as simple as adding another 4 lines to the vm.sh file in /opt/circonus/etc/node-agent.d/linux Below is the default vm.sh file with the following additions: let MEM_PERC=100*$MEM_USED/$MEM_TOTAL let SWAP_PERC=100*${SWAP[2]}/${SWAP[1]} and then 2 print statements: print_vm memory perc $MEM_PERC print_vm swap perc $SWAP_PERC #!/bin/bash print_vm() { printf "%s`%s\tL\t%s\n" $1 $2 $3 } MEM=($(free -b | grep ^Mem:)) MEM_TOTAL=${MEM[1]} # For consistency across platforms, count cache as free, not used let MEM_USED=${MEM[1]}-${MEM[3]}-${MEM[5]}-${MEM[6]} let MEM_FREE=${MEM[3]}+${MEM[5]}+${MEM[6]} let MEM_PERC=100*$MEM_USED/$MEM_TOTAL SWAP=($(free -b | grep ^Swap:)) SWAP_TOTAL=${SWAP[1]} SWAP_USED=${SWAP[2]} SWAP_FREE=${SWAP[3]} let SWAP_PERC=100*${SWAP[2]}/${SWAP[1]} # pgfault is min+maj PG_FAULTS=$(grep ^pgfault /proc/vmstat | awk '{ print $2 }') PG_MAJFAULTS=$(grep ^pgmajfault /proc/vmstat | awk '{ print $2 }') let PG_MINFAULTS=$PG_FAULTS-$PG_MAJFAULTS print_vm memory total $MEM_TOTAL print_vm memory used $MEM_USED print_vm memory free $MEM_FREE print_vm memory perc $MEM_PERC print_vm swap total $SWAP_TOTAL print_vm swap used $SWAP_USED print_vm swap free $SWAP_FREE print_vm swap perc $SWAP_PERC print_vm info page_fault $PG_FAULTS print_vm info page_fault`minor $PG_MINFAULTS print_vm info page_fault`major $PG_MAJFAULTS Just remember to restart nad (sudo /etc/init.d/nad restart), then go to the existing check and enable the 2 new metrics! You can also change the 100 to 10,000 above, turning the resulting %age into a 4 digit number, then apply =VAL/100 to the metric in the graph to get XX.XX% for more precision.

On a Linux box - If you would like to see a metric with percentage memory and swap used, it's as simple as adding another 4 lines to the vm.sh file in /opt/circonus/etc/node-agent.d/linux Below is the default vm.sh file with the following additions: <pre style="padding-top: 2px; padding-right: 2px; padding-left: 15px; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; margin-bottom: 20px; border: 1px solid rgba(0, 0, 0, 0.14902); background-color: rgb(245, 245, 245);"><ol style="padding-left: 0px; margin-top: 0px; margin-bottom: 0px; margin-left: 25px;"> <li> let MEM_PERC=100*$MEM_USED/$MEM_TOTAL </li> <li style="background: rgb(238, 238, 238);"> let SWAP_PERC=100*${SWAP[2]}/${SWAP[1]} </li> </ol></pre> and then 2 print statements: <pre style="padding-top: 2px; padding-right: 2px; padding-left: 15px; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; margin-bottom: 20px; border: 1px solid rgba(0, 0, 0, 0.14902); background-color: rgb(245, 245, 245);"><ol style="padding-left: 0px; margin-top: 0px; margin-bottom: 0px; margin-left: 25px;"> <li>print_vm memory perc $MEM_PERC</li> <li style="background: rgb(238, 238, 238);">print_vm swap perc $SWAP_PERC</li> </ol></pre> <pre style="padding-top: 2px; padding-right: 2px; padding-left: 15px; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; margin-bottom: 20px; border: 1px solid rgba(0, 0, 0, 0.14902); background-color: rgb(245, 245, 245);"><ol style="padding-left: 0px; margin-top: 0px; margin-bottom: 0px; margin-left: 25px;"> <li>#!/bin/bash</li> <li style="background: rgb(238, 238, 238);"> print_vm() { </li> <li> printf "%s`%s\tL\t%s\n" $1 $2 $3 </li> <li style="background: rgb(238, 238, 238);">}</li> <li> MEM=($(free -b | grep ^Mem:)) </li> <li style="background: rgb(238, 238, 238);"> MEM_TOTAL=${MEM[1]} </li> <li># For consistency across platforms, count cache as free, not used</li> <li style="background: rgb(238, 238, 238);"> let MEM_USED=${MEM[1]}-${MEM[3]}-${MEM[5]}-${MEM[6]} </li> <li> let MEM_FREE=${MEM[3]}+${MEM[5]}+${MEM[6]} </li> <li style="background: rgb(238, 238, 238);"> let MEM_PERC=100*$MEM_USED/$MEM_TOTAL </li> <li> SWAP=($(free -b | grep ^Swap:)) </li> <li style="background: rgb(238, 238, 238);"> SWAP_TOTAL=${SWAP[1]} </li> <li> SWAP_USED=${SWAP[2]} </li> <li style="background: rgb(238, 238, 238);"> SWAP_FREE=${SWAP[3]} </li> <li> let SWAP_PERC=100*${SWAP[2]}/${SWAP[1]} </li> <li style="background: rgb(238, 238, 238);"># pgfault is min+maj</li> <li> PG_FAULTS=$(grep ^pgfault /proc/vmstat | awk '{ print $2 }') </li> <li style="background: rgb(238, 238, 238);"> PG_MAJFAULTS=$(grep ^pgmajfault /proc/vmstat | awk '{ print $2 }') </li> <li> let PG_MINFAULTS=$PG_FAULTS-$PG_MAJFAULTS </li> <li style="background: rgb(238, 238, 238);">print_vm memory total $MEM_TOTAL</li> <li>print_vm memory used $MEM_USED</li> <li style="background: rgb(238, 238, 238);">print_vm memory free $MEM_FREE</li> <li>print_vm memory perc $MEM_PERC</li> <li style="background: rgb(238, 238, 238);">print_vm swap total $SWAP_TOTAL</li> <li>print_vm swap used $SWAP_USED</li> <li style="background: rgb(238, 238, 238);">print_vm swap free $SWAP_FREE</li> <li>print_vm swap perc $SWAP_PERC</li> <li style="background: rgb(238, 238, 238);">print_vm info page_fault $PG_FAULTS</li> <li> print_vm info page_fault`minor $PG_MINFAULTS </li> <li style="background: rgb(238, 238, 238);"> print_vm info page_fault`major $PG_MAJFAULTS </li> </ol></pre> Just remember to restart nad (sudo /etc/init.d/nad restart), then go to the existing check and enable the 2 new metrics! You can also change the 100 to 10,000 above, turning the resulting %age into a 4 digit number, then apply =VAL/100 to the metric in the graph to get XX.XX% for more precision.

6000171359

74 6000066879

2015-09-24T16:23:43-04:00

6003235491

Adding %age memory and swap used to nad

2015-09-24T16:04:02-04:00

6003235491

2015-09-24T16:27:00-04:00

WARNING: BEACONS ARE A DEPRECATED FEATURE. This information is maintained for legacy purposes. Beacons are Circonus's way of dealing with multi-dimensional real-time data. Typical metric data comes in as "just numbers" over which Circonus can build time-windowed statistical aggregates, or histograms. Occasionally, you have data that is simply too complex for this simple treatment. A common web-based example and then something more systems oriented will help illustrate the type of information Beacons can handle. Web-based Example # Each time a user visits your website, there is a variety of information available regarding both the user's system, location, and interest, as well as telemetry such as performance characteristics. Concrete examples of each are the type and version of their browser, their IP address, geographic location and ISP, the page they are viewing, the referring site, and all of the measurements from the web timing specification. Using Beacons, you can collect this information from visitors and apply real-time, complex queries to that data. Database Query Example # Circonus Beacons do not care about what the specific dimensions of your data are. While in the above example we use dimensions very specific to web browsing, we could instead use database specific dimensions. Using beacons we can track every database request, the type of request (SELECT/UPDATE/DELETE), the client that issued the request, the number of tuples involved and the duration of the statement.

WARNING: BEACONS ARE A DEPRECATED FEATURE. This information is maintained for legacy purposes. Beacons are Circonus's way of dealing with multi-dimensional real-time data. Typical metric data comes in as "just numbers" over which Circonus can build time-windowed statistical aggregates, or histograms. Occasionally, you have data that is simply too complex for this simple treatment. A common web-based example and then something more systems oriented will help illustrate the type of information Beacons can handle. Web-based Example # Each time a user visits your website, there is a variety of information available regarding both the user's system, location, and interest, as well as telemetry such as performance characteristics. Concrete examples of each are the type and version of their browser, their IP address, geographic location and ISP, the page they are viewing, the referring site, and all of the measurements from the web timing specification. Using Beacons, you can collect this information from visitors and apply real-time, complex queries to that data. Database Query Example # Circonus Beacons do not care about what the specific dimensions of your data are. While in the above example we use dimensions very specific to web browsing, we could instead use database specific dimensions. Using beacons we can track every database request, the type of request (SELECT/UPDATE/DELETE), the client that issued the request, the number of tuples involved and the duration of the statement.

6000171359

102 6000066894

2015-10-23T11:10:01-04:00

6003235491

Realtime Data Input - Beacons

2015-09-24T16:27:00-04:00

6003235491

2015-09-24T16:41:29-04:00

Data Collection Circonus brokers have the ability to both actively collect and passively receive telemetry data from systems. Active Collection Active collection (also known as polling) is when the broker plays the active role in collecting data from a system. This usually is as simple as the broker asking a system a question and then waiting for the answer(s). Circonus's architecture is specifically designed to overcome scaling challenges often present in polling-based monitoring systems. Circonus can scale to hundreds of thousands of polling checks without issue. Passive Collection The opposite of active collection is when the broker plays a passive role in data collection. In this scenario, the broker collects telemetry data from the system being monitored as it is emitted. Choosing Active vs. Passive Circonus's design eliminates the scaling challenges associated with active data collection, allowing users to choose the method that offers the most value on the data in question. The decision usually depends on data velocity and observability. If the data you are interested in is occurring quickly and persistently (a high maintained velocity), then it is often the case that you want to assess the system by observing actual work. This would mean we should use a passive monitoring setup to collect such data. However, not all systems expose this data in a way that can be observed passively, so you are limited to systems that expose data in this way (over protocols such as statsd, collectd, or HTTPtrap). ACTIVE VS PASSIVE DATABASE EXAMPLE For example, say you would like to monitor a MySQL database server and must decide whether to use Active or Passive Collection. Ideally you would like to know how many transactions per second are ongoing. The way tools typically do this is: (1) contact the server and ask how many total transactions have been performed, (2) wait, (3) contact the server and ask again. By subtracting the the two results and dividing the difference by the elapsed time between queries we can determine how many queries/second were executed during that time period. This is an active collection method. Passive collection of the same database requires the observation of every query in the system and the submission of its latency. This is possible within Circonus for some databases. For example, the PostgreSQL database has a plugin that will emit telemetry data regarding every query via the statsd protocol. Circonus can be configured to receive this statsd data and provide rich and wholly-representative information about query latencies over time using histograms. Note Some systems will deploy an agent on the database server that actively checks the instance at regular intervals and then passively submits the data to a monitoring system. This hybrid approach (while workable) is inappropriate for Circonus. It is intended to avoid problems of scalability that are simply not present in Circonus and removes the value of real-time, on-demand data collection.

<h1>Data Collection</h1> <div>Circonus brokers have the ability to both actively collect and passively receive telemetry data from systems. </div> <div> </div> <div> <h2>Active Collection</h2> </div> <div> </div> <div>Active collection (also known as polling) is when the broker plays the active role in collecting data from a system. This usually is as simple as the broker asking a system a question and then waiting for the answer(s). </div> <div> </div> <div>Circonus's architecture is specifically designed to overcome scaling challenges often present in polling-based monitoring systems. Circonus can scale to hundreds of thousands of polling checks without issue. </div> <div> </div> <div> <h2>Passive Collection</h2> </div> <div> </div> <div>The opposite of active collection is when the broker plays a passive role in data collection. In this scenario, the broker collects telemetry data from the system being monitored as it is emitted. </div> <div> </div> <div> <h2>Choosing Active vs. Passive</h2> </div> <div> </div> <div>Circonus's design eliminates the scaling challenges associated with active data collection, allowing users to choose the method that offers the most value on the data in question.</div> <div> </div> <div>The decision usually depends on data velocity and observability. If the data you are interested in is occurring quickly and persistently (a high maintained velocity), then it is often the case that you want to assess the system by observing actual work. This would mean we should use a passive monitoring setup to collect such data. However, not all systems expose this data in a way that can be observed passively, so you are limited to systems that expose data in this way (over protocols such as statsd, collectd, or HTTPtrap). </div> <div> </div> <div> <h3>ACTIVE VS PASSIVE DATABASE EXAMPLE</h3> </div> <div> </div> <div>For example, say you would like to monitor a MySQL database server and must decide whether to use Active or Passive Collection. Ideally you would like to know how many transactions per second are ongoing. The way tools typically do this is: (1) contact the server and ask how many total transactions have been performed, (2) wait, (3) contact the server and ask again. By subtracting the the two results and dividing the difference by the elapsed time between queries we can determine how many queries/second were executed during that time period. This is an active collection method.</div> <div>Passive collection of the same database requires the observation of every query in the system and the submission of its latency. This is possible within Circonus for some databases. For example, the PostgreSQL database has a plugin that will emit telemetry data regarding every query via the statsd protocol. Circonus can be configured to receive this statsd data and provide rich and wholly-representative information about query latencies over time using histograms. </div> <div> </div> <div> <h2>Note</h2> </div> <div> </div> <div>Some systems will deploy an agent on the database server that actively checks the instance at regular intervals and then passively submits the data to a monitoring system. This hybrid approach (while workable) is inappropriate for Circonus. It is intended to avoid problems of scalability that are simply not present in Circonus and removes the value of real-time, on-demand data collection.</div> <div> </div> <div> </div>

6000171359

177 6000066899

2015-10-29T21:25:48-04:00

6003235491

How is Data Collected in Circonus?

2015-09-24T16:41:29-04:00

6003235491

2015-09-24T17:32:20-04:00

Removing all brokers from the check bundle is the current way to disable it. Once disabled, the historical data can be viewed, but this won't count towards your metric limit on your account.

Removing all brokers from the check bundle is the current way to disable it. Once disabled, the historical data can be viewed, but this won't count towards your metric limit on your account.

6000171359

71 6000066924

2015-10-23T13:46:02-04:00

6003235491

How do I disable a Check?

2015-09-24T17:32:20-04:00

6003235491

2015-12-14T13:00:24-05:00

Timing on Composite Metrics Setting the timing for composite metrics can be tricky, because you can't expect a one-to-one ratio of composite values to check values. That won't happen because of differences in how the data is collected. Composite values smooth out over time. For example, say you want to try to produce that one-to-one ratio, and you're dealing with data that is sent once a minute, so you use an average value over 60 seconds. The values won't match up. This is because of variance in when the check fires. Sometimes, there will be two values within the same 60 second period, even if data is only being sent once a minute. The second value comes in just under the wire. If you never want see that happen, you can set the time to 59 seconds. That way, you'll never get two values in the same period, but you will sometimes get a zero value. This may also be undesirable, since it will likely trigger an alert. If that's an issue for you, you'll need to up the time to above 59 seconds, and you're back where you started. You can't win. Generally speaking, it is preferable to use a larger time period (3-5 minutes) and just be aware that you're pulling a larger average. If you're just using a composite metric for basic alerting, you can set a shorter time period (around 90 seconds) and alert on that. However, it can't be too short because if you alert on an interval of 59 seconds or less with no delay, you're going to get pinged constantly.

<div dir="ltr"> <div> <h2>Timing on Composite Metrics</h2> Setting the timing for composite metrics can be tricky, because you can't expect a one-to-one ratio of composite values to check values. That won't happen because of differences in how the data is collected. Composite values smooth out over time. For example, say you want to try to produce that one-to-one ratio, and you're dealing with data that is sent once a minute, so you use an average value over 60 seconds. The values won't match up. This is because of variance in when the check fires. Sometimes, there will be two values within the same 60 second period, even if data is only being sent once a minute. The second value comes in just under the wire. If you never want see that happen, you can set the time to 59 seconds. That way, you'll never get two values in the same period, but you will sometimes get a zero value. This may also be undesirable, since it will likely trigger an alert. If that's an issue for you, you'll need to up the time to above 59 seconds, and you're back where you started. You can't win. Generally speaking, it is preferable to use a larger time period (3-5 minutes) and just be aware that you're pulling a larger average. If you're just using a composite metric for basic alerting, you can set a shorter time period (around 90 seconds) and alert on that. However, it can't be too short because if you alert on an interval of 59 seconds or less with no delay, you're going to get pinged constantly. </div> </div>

6000171359

102 6000098267

2016-01-29T18:22:36-05:00

6003235491

A Primer on Composite Metrics

2016-01-29T18:22:36-05:00

6002695175

2015-09-24T18:01:39-04:00

With our Circonus-Inside version, you can store the data indefinitely. This is based on the amount of storage that you have allocated. The larger the amount of data and/or longer the retention period the more storage is required.

With our Circonus-Inside version, you can store the data indefinitely. This is based on the amount of storage that you have allocated. The larger the amount of data and/or longer the retention period the more storage is required.

6000171359

50 6000066938

2015-09-24T18:01:39-04:00

6003235491

What is the storage time period in Circonus-Inside?

2015-09-24T18:01:39-04:00

6003235491

2015-09-24T18:18:57-04:00

Often, users of Circonus will add the first few checks by hand in the UI. The web interface is great for seeing all available check types at a glance and easily modifying various aspects of a new check. However, once you have an idea of what you'd like to create, you need to scale that out to your entire infrastructure. Typically, our customers will use one of the following options: Templates - Circonus templates are great if you have lots of checks that are going to be exactly the same, such as DNS or database checks. Rulesets on the master check can also be synced to the child checks, making it easy to apply lots of checks across your infrastructure. API - If you're going to have slight modifications to each check, or need a little more logic built into the check creation, then the API is for you. The extensive Circonus REST API will allow you to create and modify a lot of checks at once while reading from your CMDB or configuration repository. If you plan on scanning your network for all devices that are available, the API can help you create checks as you find new devices, based on which ports are open and what is listening on those ports. Chef, Puppet, or Ansible - If you've got a configuration management system already in place, you can utilize the Circonus API to create checks, graphs, and rulesets as a part of your provisioning process. This is great for dynamic environments, as you can bring up new checks when the server first comes online, while utilizing tags to help organize your systems. Node agent auto-registration - If you're using nad as your agent to get system information from your servers, there is a command-line flag for registering that server's checks and graphs. This allows you to set the configuration for your servers ahead of time, then have each register itself when nad is ready. -kjohnson

Often, users of Circonus will add the first few checks by hand in the UI. The web interface is great for seeing all available check types at a glance and easily modifying various aspects of a new check. However, once you have an idea of what you'd like to create, you need to scale that out to your entire infrastructure. Typically, our customers will use one of the following options: <ol> <li>Templates - Circonus templates are great if you have lots of checks that are going to be exactly the same, such as DNS or database checks. Rulesets on the master check can also be synced to the child checks, making it easy to apply lots of checks across your infrastructure. </li> <li>API - If you're going to have slight modifications to each check, or need a little more logic built into the check creation, then the API is for you. The extensive Circonus REST API will allow you to create and modify a lot of checks at once while reading from your CMDB or configuration repository. If you plan on scanning your network for all devices that are available, the API can help you create checks as you find new devices, based on which ports are open and what is listening on those ports. </li> <li>Chef, Puppet, or Ansible - If you've got a configuration management system already in place, you can utilize the Circonus API to create checks, graphs, and rulesets as a part of your provisioning process. This is great for dynamic environments, as you can bring up new checks when the server first comes online, while utilizing tags to help organize your systems. </li> <li>Node agent auto-registration - If you're using nad as your agent to get system information from your servers, there is a command-line flag for registering that server's checks and graphs. This allows you to set the configuration for your servers ahead of time, then have each register itself when nad is ready. </li> </ol> -kjohnson

6000171359

48 6000066948

2015-10-23T14:24:37-04:00

6003235491

What options exist to add many checks at once?

2015-09-24T18:18:57-04:00

6003235491

2015-09-24T18:14:59-04:00

There is a great group on Linkedin, the Big Data and Analytics Grou7p: https://www.linkedin.com/groups?home=&gid=4332669&trk=groups_item_detail-h-logo3 With an interesting post - 20 short tutorials all data scientists should read (and practice): https://www.linkedin.com/today/post/article/20140805085133-52688293-20-short-tutorials-all-data-scientists-should-read-and-practice1

There is a great group on Linkedin, the Big Data and Analytics Grou7p: <a href="https://www.linkedin.com/groups?home=&gid=4332669&trk=groups_item_detail-h-logo" style="color: rgb(59, 115, 185);">https://www.linkedin.com/groups?home=&gid=4332669&trk=groups_item_detail-h-logo3</a> With an interesting post - 20 short tutorials all data scientists should read (and practice): <a href="https://www.linkedin.com/today/post/article/20140805085133-52688293-20-short-tutorials-all-data-scientists-should-read-and-practice" style="color: rgb(59, 115, 185);">https://www.linkedin.com/today/post/article/20140805085133-52688293-20-short-tutorials-all-data-scientists-should-read-and-practice1</a>

6000171359

28 6000066944

2015-10-30T11:15:06-04:00

6003235491

Some Interesting Resources

2015-10-30T11:15:04-04:00

6003235491

2015-09-24T18:29:21-04:00

Installing NAD First, install NAD by either building it yourself or using a pre-built package. Here's an example of building it yourself (done on CentOS): % sudo yum install epel-release % sudo yum install nodejs % wget https://github.com/circonus-labs/nad/archive/master.zip % unzip master.zip % cd nad-master/ % sudo make install-rhel % sudo /etc/init.d/nad start Here's an example of installing our pre-built packages (look for your OS and 32 or 64 bit version): On Centos: % sudo rpm -i http://updates.circonus.net/node-agent/packages/YOUR_SYSTEM % sudo /etc/init.d/nad start On Ubuntu % wget http://updates.circonus.net/node-agent/packages/YOUR_SYSTEM % dpkg -i YOUR_SYSTEM_nad_file.deb Then test it: % curl http://localhost:2609/ How to extend Nad? Nad runs a series of scripts, located in /opt/circonus/etc/node-agent.d. If you want to extend, check out https://github.com/circonus-labs/nad/blob/master/nad.8 Beginning on line 79, there is a description summarizing the format that the scripts should be in. If you want to modify the checks, or create your own, you can either edit the existing scripts or add your own script there. Then, when you go back to the UI, you will see your changes or additions automatically.

<h3>Installing NAD</h3> First, install NAD by either building it yourself or using a pre-built package. Here's an example of building it yourself (done on CentOS): <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"> % sudo yum install epel-release % sudo yum install nodejs % wget https://github.com/circonus-labs/nad/archive/master.zip % unzip master.zip % cd nad-master/ % sudo make install-rhel % sudo /etc/init.d/nad start </blockquote> Here's an example of installing our pre-built packages (look for your OS and 32 or 64 bit version): On Centos: <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"> % sudo rpm -i http://updates.circonus.net/node-agent/packages/YOUR_SYSTEM % sudo /etc/init.d/nad start </blockquote> On Ubuntu <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"> % wget http://updates.circonus.net/node-agent/packages/YOUR_SYSTEM % dpkg -i YOUR_SYSTEM_nad_file.deb </blockquote> Then test it: <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;">% curl http://localhost:2609/ </blockquote> <h3>How to extend Nad? </h3> Nad runs a series of scripts, located in /opt/circonus/etc/node-agent.d. If you want to extend, check out <a href="https://github.com/circonus-labs/nad/blob/master/nad.8" target="" style="line-height: 16px;">https://github.com/circonus-labs/nad/blob/master/nad.8</a> Beginning on line 79, there is a description summarizing the format that the scripts should be in. If you want to modify the checks, or create your own, you can either edit the existing scripts or add your own script there. Then, when you go back to the UI, you will see your changes or additions automatically.

6000171359

113 6000066956

2015-10-29T21:53:31-04:00

6003235491

How to use/extend node agent?

2015-09-24T18:29:51-04:00

6003235491

2015-09-24T18:33:16-04:00

Circonus is a telemetry collecting platform. It is not focused on log data, nor on documents. Each piece of data should be concise and representative of the metric to which it is attached. Collected data is broken down into four types: numeric, text, histogram, and beacons. TEXT Data Text data in Circonus is a brief, terse measurement that is designed to express a system's current state. The values are limited to 512 bytes of data. The typical uses for the text data type are SSH fingerprints, SSL certificate subjects, software version numbers, and response codes (such as HTTP response codes). For those not familiar with HTTP response codes, they are actually restricted by the numeric range in the HTTP specification, but interpreting them as numbers makes no analytical sense. Each response code (and range) has unique meaning and thus using statistical aggregates (such as mean or stddev) would be nonsensical. NUMERIC Data Numeric data is as simple as it sounds: a number. Examples include the number of visitors to a website, the number of database transactions executed, the amount of traffic push from a network interface, the revenues, the number of bytes used on a disk, or the temperature outside. Numeric types in Circonus have no units (e.g. visitors, transaction, octets, $USD, bytes, Celsius, etc.). Unit information is not included as there are no limits on the type of units in which numbers may be submitted. In the event that the units are not self evident from the raw metric name, it is advisable to name metrics in the system such that the base units are obvious. Common convention is to include a suffix, such as "_dollars", "_visitors", or "_ms" for milliseconds. When Circonus stores numeric data, it aggregates collected data into one minute windows. Within this window, we track: count - the total number of sample, average/gauge - the average of the samples stddev - the standard deviation across the samples derive - the first order derivative over the samples derive_stddev - the standard deviation over the 'derive' samples counter - the first order derivative ignoring decreases in sample value counter_stddev - the standard deviation over the 'counter' samples The "counter" type is used to determine the rate of change of telemetry data you know is monotonically increasing, but due to technical details can "reset" to a lower value (often zero). If you know that a rate can never legitimately be negative (such as bandwidth or arrivals/second or transactions/second), then the "counter" aggregate is the interesting aggregate of the bunch. HISTOGRAM Data Histograms are designed for understand high velocity data. Histogram data will store all samples reduced to two significant digits in base ten. (E.g. 0.1 through 9.9 at increments of 0.1, 10 through 99 at increments of 1, 100 through 990 at increments of 10, etc.) It handles values from (+/-)10^127 to (+/-)10^-128 and zero. The histogram type stores the counts of samples in one minute time chunks (or "buckets"). Statistical information can be calculated from the histogram with error introduced from the bucketing. BEACONS Beacon data is the one data type within Circonus that is not stored permanently. Beacons are deprecated feature, and are no longer used, except for legacy purposes.

<div> Circonus is a telemetry collecting platform. It is not focused on log data, nor on documents. Each piece of data should be concise and representative of the metric to which it is attached. Collected data is broken down into four types: numeric, text, histogram, and beacons. <h3 style="margin-top: 10px; margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 24.5px;">TEXT Data</h3> Text data in Circonus is a brief, terse measurement that is designed to express a system's current state. The values are limited to 512 bytes of data. The typical uses for the text data type are SSH fingerprints, SSL certificate subjects, software version numbers, and response codes (such as HTTP response codes). For those not familiar with HTTP response codes, they are actually restricted by the numeric range in the HTTP specification, but interpreting them as numbers makes no analytical sense. Each response code (and range) has unique meaning and thus using statistical aggregates (such as mean or stddev) would be nonsensical. <h3 style="margin-top: 10px; margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 24.5px;">NUMERIC Data</h3> Numeric data is as simple as it sounds: a number. Examples include the number of visitors to a website, the number of database transactions executed, the amount of traffic push from a network interface, the revenues, the number of bytes used on a disk, or the temperature outside. Numeric types in Circonus have no units (e.g. visitors, transaction, octets, $USD, bytes, Celsius, etc.). Unit information is not included as there are no limits on the type of units in which numbers may be submitted. In the event that the units are not self evident from the raw metric name, it is advisable to name metrics in the system such that the base units are obvious. Common convention is to include a suffix, such as "_dollars", "_visitors", or "_ms" for milliseconds. When Circonus stores numeric data, it aggregates collected data into one minute windows. Within this window, we track: <li style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">count - the total number of sample,</li> <li style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">average/gauge - the average of the samples</li> <li style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">stddev - the standard deviation across the samples</li> <li style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">derive - the first order derivative over the samples</li> <li style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">derive_stddev - the standard deviation over the 'derive' samples</li> <li style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">counter - the first order derivative ignoring decreases in sample value</li> <li style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">counter_stddev - the standard deviation over the 'counter' samples</li> The "counter" type is used to determine the rate of change of telemetry data you know is monotonically increasing, but due to technical details can "reset" to a lower value (often zero). If you know that a rate can never legitimately be negative (such as bandwidth or arrivals/second or transactions/second), then the "counter" aggregate is the interesting aggregate of the bunch. <h3 style="margin-top: 10px; margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 24.5px;">HISTOGRAM Data</h3> Histograms are designed for understand high velocity data. Histogram data will store all samples reduced to two significant digits in base ten. (E.g. 0.1 through 9.9 at increments of 0.1, 10 through 99 at increments of 1, 100 through 990 at increments of 10, etc.) It handles values from (+/-)10^127 to (+/-)10^-128 and zero. The histogram type stores the counts of samples in one minute time chunks (or "buckets"). Statistical information can be calculated from the histogram with error introduced from the bucketing. <h3 style="margin-top: 10px; margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 24.5px;">BEACONS</h3> Beacon data is the one data type within Circonus that is not stored permanently. Beacons are deprecated feature, and are no longer used, except for legacy purposes. </div>

6000171359

300 6000066958

2016-05-15T17:40:40-04:00

6003235491

Data Types in Circonus - Text, Numeric, Histogram, and Beacons

2016-05-15T17:40:40-04:00

6003235491

2015-09-25T17:54:36-04:00

My PostgreSQL check against pgbouncer used to work and now it doesn’t. How do I fix it? If you have set up a PostgreSQL check against pgbouncer (rather than postgres) and it used to work and now it doesn’t it, might be because Circonus switched out the postgres client implementation to Java (for better stability). To fix this problem, users should add the following line to their pgbouncer.conf file: ignore_startup_parameters = extra_float_digits

My PostgreSQL check against pgbouncer used to work and now it doesn’t. How do I fix it? If you have set up a PostgreSQL check against pgbouncer (rather than postgres) and it used to work and now it doesn’t it, might be because Circonus switched out the postgres client implementation to Java (for better stability). To fix this problem, users should add the following line to their pgbouncer.conf file: <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;">ignore_startup_parameters = extra_float_digits</blockquote>

6000171359

41 6000067382

2015-10-30T11:33:38-04:00

6003235491

PostgreSQL check against pgbouncer

2015-10-30T11:33:07-04:00

6003235491

2015-09-24T18:34:33-04:00

Check Type Guides: NOTE: This page DOES NOT contain all the Checks that available in Circonus. All the Check types available in Circonus are documented in the User Manual. but these Overview Guides provide detailed step-by-step instructions for specific Check types. If more Check Guides are created, they will be added to the list. If a Check you are interested in is not yet listed, contact us! This article lists just a few of the Checks that are available in Circonus. The PDF files that are listed below provides you with an overview of the Check with easy to follow instructions on which buttons to click, the order that data should be entered into the form, and information on the what the fields are and what could be entered into them. If you use the API (see the API documentation) for creating Checks, then these documents can be used to give you some insight into what are the required and optional fields for the Check, thereby making your coding for the API easier to create. The following is the list of available Check Guides: Adding_a_Check (Overview) Cloudwatch Elasticsearch External Check HTTP NAD Agent (RHEL/Centos, Ubuntu, Illumos (SmartOS, OmniTI, OpenIndiana, etc.) PING Resmon SNMP SSH Windows Agent (requires .NET 4.0) Windows NAD Agent -cmadsen

Check Type Guides:NOTE: This page DOES NOT contain all the Checks that available in Circonus. All the Check types available in Circonus are documented in the <a href="https://login.circonus.com/user/docs/Data/CheckTypes" target="">User Manual</a>. but these Overview Guides provide detailed step-by-step instructions for specific Check types. If more Check Guides are created, they will be added to the list. If a Check you are interested in is not yet listed, contact us!This article lists just a few of the Checks that are available in Circonus. The PDF files that are listed below provides you with an overview of the Check with easy to follow instructions on which buttons to click, the order that data should be entered into the form, and information on the what the fields are and what could be entered into them.If you use the API (see the <a href="6000055852-the-circonus-api">API documentation</a>) for creating Checks, then these documents can be used to give you some insight into what are the required and optional fields for the Check, thereby making your coding for the API easier to create.The following is the list of available Check Guides: <blockquote style="margin: 0px 0px 0px 40px; border: none;"> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/Adding_a_Check.pdf" style="color: rgb(59, 115, 185);">Adding_a_Check (Overview)</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/Cloudwatch_js.pdf" style="color: rgb(59, 115, 185);">Cloudwatch</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/Elasticsearch_ja.pdf" style="color: rgb(59, 115, 185);">Elasticsearch</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/external_ja.pdf" style="color: rgb(59, 115, 185);">External Check</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/HTTP_ja.pdf" style="color: rgb(59, 115, 185);">HTTP</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/Node_agent_ja.pdf" style="color: rgb(59, 115, 185);">NAD Agent (RHEL/Centos, Ubuntu, Illumos (SmartOS, OmniTI, OpenIndiana, etc.)</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/ping_ja.pdf" style="color: rgb(59, 115, 185);">PING</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/resmon_ja.pdf" style="color: rgb(59, 115, 185);">Resmon</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/snmp_ja.pdf" style="color: rgb(59, 115, 185);">SNMP</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/ssh2_ja.pdf" style="color: rgb(59, 115, 185);">SSH</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/Windows_agent_ja.pdf" style="color: rgb(59, 115, 185);">Windows Agent (requires .NET 4.0)</a> <img src="http://www.circonus.com/i/pdf.gif" alt="" class="fr-dii fr-draggable"> <a href="http://www.circonus.com/files/checks/Windows_node_ja.pdf" style="color: rgb(59, 115, 185);">Windows NAD Agent</a> </blockquote>-cmadsen

6000171359

82 6000066959

2017-09-20T17:03:09-04:00

6003235491

Check Type Overview Guides

2015-09-24T18:34:33-04:00

6003235491

2015-09-24T18:37:24-04:00

Our HTTP check looks at the body of the URL you give it, and will return things like HTTP code, duration for the request, time to connect and receive the first byte, as well as a plethora of SSL information if applicable. As for transactions, you can send content or headers with the request if that works for your case, but you can not tell it to click a button or a link, etc. We also support KeyNote - which does quite a bit more via their awesome XML Data Pulse API.

Our HTTP check looks at the body of the URL you give it, and will return things like HTTP code, duration for the request, time to connect and receive the first byte, as well as a plethora of SSL information if applicable. As for transactions, you can send content or headers with the request if that works for your case, but you can not tell it to click a button or a link, etc. We also support <a href="http://www.circonus.com/exploring-keynote-xml-data-pulse/" target="">KeyNote</a> - which does quite a bit more via their awesome XML Data Pulse API.

6000171359

17 6000066960

2015-10-23T14:20:30-04:00

6003235491

What are your website monitoring capabilities?

2015-09-24T18:37:24-04:00

6003235491

2016-03-25T10:57:48-04:00

Circonus supports data ingestion from a wide variety of sources in all kinds of formats. This data is transformed and processed internally and ultimately stored in the form of 1M-rollup-aggregates, that come in two flavors (cf. DataTypes): 1. Numeric metrics, which store a number of summary statistics, including the mean value across the rollup-window. 2. Histogram metrics, which store the sample distribution as key-value pairs (Histogram Representation). This note explains how these rollup-aggregates are derived from the incoming raw-data. 1. Rolling-up Numeric Data We will cover the most common case first, that raw numeric values are submitted to Circonus, one at a time. The Numeric raw values r that arrive at Circonus have an associated timestamp, r.t: long, which is a (long) integer in milliseconds, and an associated value, r.v: float. The stream of incoming raw values is divided into rollup windows, w, of equal duration, which is the rollup period, p. Currently, the minimum supported rollup period is 1 minute (p = 60000). The rollup windows carry a start time, w.start, that is a multiple of the rollup period. The window end time is defined to be w.end = w.start + p. A raw value, r, belongs to a rollup window, w, if: w.start < r.t and r.t < w.end The rollup process takes a list of raw-values, R, that belong to a rollup window, w, and are sorted in order of the time-stamp, and computes the following of summary statistics over the raw values: w.count is the number of samples in R: w.count = #R w.value is the average value of all sample values in R: w.stddev is the sample standard devation of all samples in R. Requires: #R > 2 w.derivative is the effective rate of change of the samples per second. It is defined as: Note, that there are 1000 milliseconds in one second. There is an alternative way to compute w.derivative that will be used later on. It is based on the notion of the difference series DT and DV of R which are: DV[i] = R[i+1].v - R[i].v DT[i] = R[i+1].t - R[i].t Here is the alternative computation: w.derivative_stddev is defined as the (uncorrected) standard derivation of the difference series DV weighted by DT: The weighted standard deviation is defined as: w.counter, the counter derivative rollup statistics, is similar to the derivative rollup, but is suitable to raw value streams that experience overflow events that should be ignored by the metric. Examples for such streams include the number of octects observed by a network switch, or the total visit count of a server that gets restarted occasionally. To define the counter derivative rollup, we use the DV and DT series from above. Let CV and CT be copies of DV and DT where indices i with DV[i] < 0 are removed from both series. Then: w.counter_stddev. In analogy to w.derivative_stddev, this rollup is defined as the (uncorrected) standard derivation of the monotone difference series CV weighted by CT: w.counter_derivative_stddev = stddev(CV,CT) In addition to those numeric rollup-aggregates, Circonus also supports rollup to Histogram metrics (cf. Histograms, Histogram Internals). For numeric input values those are calculated as follows: w.histogram, the histogram statistic, can represented as an associative array that maps bucket boundaries to sample counts: here b represents a bucket boundary, which can be any float value of the form b = x * 10^e, where x is one of the following float values: x = -9.9, -9.8, -9.7 .. -1.1, -1.0, 0, 1.0, 1.1, .. ,9.9 e is an integer in the following range: e = -128 .. 127 Bucket(b) is the bucket with boundary b, e.g. if b = 1, then Bucket(b) is the interval [1, 1.1). For more details, see Histogram Internals. 2. Rolling-up Histogram Data In addition to numeric input values, Circonus also accepts histogram data submitted directly via a HTTP Trap, in the form of: Raw histograms as arrays of numeric values, e.g. [1, 2.7, 5] Raw histograms in pre-bucketed form, e.g. ["H[0.1]=3", "H[11]=7"] You can mix both types of data submission with each other and with numeric raw-values as well. The resulting histogram rollup aggregates all data that was submitted for a time period by summing the counts for each bucket. Example If the following values are submitted: | t | value | |-----+------------------------| | 0 | 1 | | 100 | [1,2,3] | | 200 | ["H[1]=3", "H[5]=3"] | | 300 | 2 | | 400 | [3,4,5] | | 500 | ["H[5]=3", "H[6]=3"] | The resulting histogram for the time period [0, 1000] is, given by: H[1] = 1 + 1 + 3 + 0 + 0 + 0 = 5 H[2] = 0 + 1 + 0 + 1 + 0 + 0 = 2 H[3] = 0 + 1 + 0 + 0 + 1 + 0 = 2 H[4] = 0 + 0 + 0 + 0 + 1 + 0 = 1 H[5] = 0 + 0 + 3 + 0 + 1 + 3 = 7 H[6] = 0 + 0 + 0 + 0 + 0 + 3 = 3 Numeric Rollup of Histogram Values When pushing raw-histogram values, numeric aggregates are also performed. However, due to the details of the data collection, the resulting rollup values are often perceived as un-intuitive. In particular, they are not equivalent to pushing the containing samples individually. Hence, we make the following recommendation: WARNING: If you use raw-histogram values, ONLY the resulting histogram metric should be used for further computations! Now, that we have warned you, we will tell you what values end up in the numeric metric, when histogram values are pushed. When a Raw histogram array is pushed, the numeric metric will receive a single sample with the average value of the array. When a Raw pre-bucketed histogram is pushed, the numeric metric will receive a nil-sample. Therefore the numeric statistics count and average are are calculated as follows: w.count = 1 + 1 + 1 + 1 + 1 + 1 = 6 w.average = 1 + 2 + 0 + 2 + 4 + 0 = 9 Note that, these are different from the total count of all samples (21) and the total average (84/21=4.0). The recommended way to derive those values is to use our Analytics Query Language, CAQL. E.g. metric:histogram(...) | histogram:count() metric:histogram(...) | histogram:mean() metric:histogram(...) | histogram:stddev() CAQL would compute approximations of those respective values from the histogram itself. Appendix: Numeric Rollup Example --- Adding rollup value {t=0, v=0.0} - count : 1 - value : 0 - stddev : 0 - derivative : 0 - derivative_stddev : 0 - counter : 0 - counter_stddev : 0 --- Adding rollup value {t=100, v=1.0} - count : 2 - value : 0.5 - stddev : 0.5 - derivative : 10 - derivative_stddev : 0 - counter : 10 - counter_stddev : 0 --- Adding rollup value {t=220, v=1.0} - count : 3 - value : 0.66666666666667 - stddev : 0.47140452265739 - derivative : 4.5454545021057 - derivative_stddev : 4.979296207428 - counter : 4.5454545021057 - counter_stddev : 4.979296207428 --- Adding rollup value {t=300, v=6.0} - count : 4 - value : 2 - stddev : 2.3452079296112 - derivative : 20 - derivative_stddev : 25.980762481689 - counter : 20 - counter_stddev : 25.980762481689 --- Adding rollup value {t=400, v=0.0} - count : 5 - value : 1.6 - stddev : 2.2449944019318 - derivative : 0 - derivative_stddev : 41.306777954102 - counter : 20 - counter_stddev : 25.980762481689 (continued) --- Adding rollup value {t=500, v=1.0} - count : 6 - value : 1.5 - stddev : 2.0615527629852 - derivative : 2 - derivative_stddev : 37.161808013916 - counter : 17.5 - counter_stddev : 22.912878036499 --- Adding rollup value {t=600, v=3.0} - count : 7 - value : 1.7142857142857 - stddev : 1.9794865846634 - derivative : 5 - derivative_stddev : 34.580821990967 - counter : 18 - counter_stddev : 20.518283843994 --- Adding rollup value {t=700, v=0.0} - count : 8 - value : 1.5 - stddev : 1.9364916086197 - derivative : 0 - derivative_stddev : 34.278270721436 - counter : 18 - counter_stddev : 20.518283843994 --- Adding rollup value {t=800, v=1.0} - count : 9 - value : 1.4444444444444 - stddev : 1.8324912786484 - derivative : 1.25 - derivative_stddev : 32.234489440918 - counter : 16.666666030884 - counter_stddev : 18.966344833374 --- Adding rollup value {t=900, v=3.0} - count : 10 - value : 1.6 - stddev : 1.7999999523163 - derivative : 3.3333332538605 - derivative_stddev : 30.956954956055 - counter : 17.1428565979 - counter_stddev : 17.598121643066

Circonus supports data ingestion from a wide variety of sources in all kinds of formats. This data is transformed and processed internally and ultimately stored in the form of 1M-rollup-aggregates, that come in two flavors (cf. <a href="https://login.circonus.com/user/docs/Data/Overview#DataTypes" style="background-color: rgb(255, 255, 255);" target="">DataTypes</a>):1. Numeric metrics, which store a number of summary statistics, including the mean value across the rollup-window.2. Histogram metrics, which store the sample distribution as key-value pairs (<a href="6000066608-histogram-bin-count" style="font-size: 13px; background-color: rgb(255, 255, 255);" target="">Histogram Representation</a>).This note explains how these rollup-aggregates are derived from the incoming raw-data. <h1 style="">1. Rolling-up Numeric Data</h1> We will cover the most common case first, that raw numeric values are submitted to Circonus, one at a time. The Numeric raw values r that arrive at Circonus have an associated timestamp, r.t: long, which is a (long) integer in milliseconds, and an associated value, r.v: float. The stream of incoming raw values is divided into rollup windows, w, of equal duration, which is the rollup period, p. Currently, the minimum supported rollup period is 1 minute (p = 60000). The rollup windows carry a start time, w.start, that is a multiple of the rollup period. The window end time is defined to be w.end = w.start + p. A raw value, r, belongs to a rollup window, w, if: <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;">w.start < r.t and r.t < w.end</blockquote> The rollup process takes a list of raw-values, R, that belong to a rollup window, w, and are sorted in order of the time-stamp, and computes the following of summary statistics over the raw values: <ul> <li> w.count is the number of samples in R: w.count = #R </li> <li> w.value is the average value of all sample values in R: <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6019057831/original/Formula1.png?1459265417" class="inline-image fr-dii fr-draggable" data-id="6019057831"> </li> <li> w.stddev is the sample standard devation of all samples in R. Requires: #R > 2 <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6019057853/original/Formula2.png?1459265440" class="inline-image fr-dii fr-draggable" data-id="6019057853"> </li> <li> w.derivative is the effective rate of change of the samples per second. It is defined as: <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6019058082/original/Formula3.png?1459265559" class="inline-image fr-dii fr-draggable" data-id="6019058082"> Note, that there are 1000 milliseconds in one second. There is an alternative way to compute w.derivative that will be used later on. It is based on the notion of the difference series DT and DV of R which are: DV[i] = R[i+1].v - R[i].v </li> </ul><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"> DT[i] = R[i+1].t - R[i].t Here is the alternative computation: <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6019058141/original/Formula4.png?1459265601" class="inline-image fr-dii fr-draggable" data-id="6019058141"> </blockquote> <ul> <li> w.derivative_stddev is defined as the (uncorrected) standard derivation of the difference series DV weighted by DT: The weighted standard deviation is defined as: <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6019058192/original/Formula5.png?1459265628" class="inline-image fr-dii fr-draggable" data-id="6019058192"> </li> <li> w.counter, the counter derivative rollup statistics, is similar to the derivative rollup, but is suitable to raw value streams that experience overflow events that should be ignored by the metric. Examples for such streams include the number of octects observed by a network switch, or the total visit count of a server that gets restarted occasionally. To define the counter derivative rollup, we use the DV and DT series from above. Let CV and CT be copies of DV and DT where indices i with DV[i] < 0 are removed from both series. Then: <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6019058260/original/Formula6.png?1459265648" class="inline-image fr-dii fr-draggable" data-id="6019058260"> </li> <li> w.counter_stddev. In analogy to w.derivative_stddev, this rollup is defined as the (uncorrected) standard derivation of the monotone difference series CV weighted by CT: w.counter_derivative_stddev = stddev(CV,CT) </li> </ul><div> </div><div> </div>In addition to those numeric rollup-aggregates, Circonus also supports rollup to Histogram metrics (cf. <a href="https://login.circonus.com/user/docs/Visualization/Graphs/View/Histograms" style="line-height: 16px; background-color: rgb(255, 255, 255);" target="">Histograms</a>, <a href="6000066608-histogram-internals" style="line-height: 16px; background-color: rgb(255, 255, 255);" target="">Histogram Internals</a>). For numeric input values those are calculated as follows: <ul><li> w.histogram, the histogram statistic, can represented as an associative array that maps bucket boundaries to sample counts: <img src="https://latex.codecogs.com/gif.latex?%5Cdpi%7B150%7D%20%5Ctexttt%7Bw.histogram%5Bb%5D%7D%20%3D%20%5C%23%20%5C%7B%20i%20%5C%3B%7C%5C%3B%20%5Ctexttt%7BR%7D%5Bi%5D%20%5Cin%20%5Ctexttt%7BBucket%28b%29%7D%20%5C%7D" class="fr-dii fr-draggable"> here b represents a bucket boundary, which can be any float value of the form b = x * 10^e, where<ul> <li>x is one of the following float values: x = -9.9, -9.8, -9.7 .. -1.1, -1.0, 0, 1.0, 1.1, .. ,9.9</li> <li>e is an integer in the following range: e = -128 .. 127</li> <li>Bucket(b) is the bucket with boundary b, e.g. if b = 1, then Bucket(b) is the interval [1, 1.1). For more details, see <a href="6000066608-histogram-internals" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif; color: rgb(0, 64, 128); line-height: 16px; background-color: rgb(255, 255, 255);" target="">Histogram Internals</a>. </li> </ul> </li></ul><h1 style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif; line-height: 42.9px;">2. Rolling-up Histogram Data</h1> In addition to numeric input values, Circonus also accepts histogram data submitted directly via a <a href="https://login.circonus.com/user/docs/Data/CheckTypes#HTTPTrapJSONDocs" style="line-height: 16px; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif; color: rgb(0, 64, 128); background-color: rgb(255, 255, 255);" target="">HTTP Trap</a>, in the form of:<ul> <li>Raw histograms as arrays of numeric values, e.g. [1, 2.7, 5] </li> <li>Raw histograms in pre-bucketed form, e.g. ["H[0.1]=3", "H[11]=7"] </li> </ul> You can mix both types of data submission with each other and with numeric raw-values as well. The resulting histogram rollup aggregates all data that was submitted for a time period by summing the counts for each bucket. Example If the following values are submitted: <pre style="">| t | value | |-----+------------------------| | 0 | 1 | | 100 | [1,2,3] | | 200 | ["H[1]=3", "H[5]=3"] | | 300 | 2 | | 400 | [3,4,5] | | 500 | ["H[5]=3", "H[6]=3"] |</pre> The resulting histogram for the time period [0, 1000] is, given by: H[1] = 1 + 1 + 3 + 0 + 0 + 0 = 5H[2] = 0 + 1 + 0 + 1 + 0 + 0 = 2H[3] = 0 + 1 + 0 + 0 + 1 + 0 = 2H[4] = 0 + 0 + 0 + 0 + 1 + 0 = 1H[5] = 0 + 0 + 3 + 0 + 1 + 3 = 7H[6] = 0 + 0 + 0 + 0 + 0 + 3 = 3 <h3 style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Numeric Rollup of Histogram Values</h3> When pushing raw-histogram values, numeric aggregates are also performed. However, due to the details of the data collection, the resulting rollup values are often perceived as un-intuitive. In particular, they are not equivalent to pushing the containing samples individually. Hence, we make the following recommendation: WARNING: If you use raw-histogram values, ONLY the resulting histogram metric should be used for further computations! Now, that we have warned you, we will tell you what values end up in the numeric metric, when histogram values are pushed. <ol> <li>When a Raw histogram array is pushed, the numeric metric will receive a single sample with the average value of the array.</li> <li>When a Raw pre-bucketed histogram is pushed, the numeric metric will receive a nil-sample.</li> </ol> Therefore the numeric statistics count and average are are calculated as follows: w.count = 1 + 1 + 1 + 1 + 1 + 1 = 6 w.average = 1 + 2 + 0 + 2 + 4 + 0 = 9 Note that, these are different from the total count of all samples (21) and the total average (84/21=4.0).The recommended way to derive those values is to use our <a href="https://login.circonus.com/user/docs/caql_reference" style="background-color: rgb(255, 255, 255);" target="">Analytics Query Language, CAQL</a>. E.g. metric:histogram(...) | histogram:count() metric:histogram(...) | histogram:mean() metric:histogram(...) | histogram:stddev() CAQL would compute approximations of those respective values from the histogram itself. <h2>Appendix: Numeric Rollup Example</h2> <table border="1" cellpadding="0" cellspacing="0" id="table48191" style="border-collapse:collapse; border:1px solid #afafaf;width:100%;"><tbody><tr class="current"> <td> --- Adding rollup value {t=0, v=0.0} - count : 1 - value : 0 - stddev : 0 - derivative : 0 - derivative_stddev : 0 - counter : 0 - counter_stddev : 0 --- Adding rollup value {t=100, v=1.0} - count : 2 <div style="text-align: left;">- value : 0.5 </div> - stddev : 0.5 - derivative : 10 - derivative_stddev : 0 - counter : 10 - counter_stddev : 0 --- Adding rollup value {t=220, v=1.0} - count : 3 - value : 0.66666666666667 - stddev : 0.47140452265739 - derivative : 4.5454545021057 - derivative_stddev : 4.979296207428 - counter : 4.5454545021057 - counter_stddev : 4.979296207428 --- Adding rollup value {t=300, v=6.0} - count : 4 - value : 2 - stddev : 2.3452079296112 - derivative : 20 - derivative_stddev : 25.980762481689 - counter : 20 - counter_stddev : 25.980762481689 --- Adding rollup value {t=400, v=0.0} - count : 5 - value : 1.6 - stddev : 2.2449944019318 - derivative : 0 - derivative_stddev : 41.306777954102 - counter : 20 - counter_stddev : 25.980762481689 </td> <td> (continued) --- Adding rollup value {t=500, v=1.0} - count : 6 - value : 1.5 - stddev : 2.0615527629852 - derivative : 2 - derivative_stddev : 37.161808013916 - counter : 17.5 - counter_stddev : 22.912878036499 --- Adding rollup value {t=600, v=3.0} - count : 7 - value : 1.7142857142857 - stddev : 1.9794865846634 - derivative : 5 - derivative_stddev : 34.580821990967 - counter : 18 - counter_stddev : 20.518283843994 --- Adding rollup value {t=700, v=0.0} - count : 8 - value : 1.5 - stddev : 1.9364916086197 - derivative : 0 - derivative_stddev : 34.278270721436 - counter : 18 - counter_stddev : 20.518283843994 --- Adding rollup value {t=800, v=1.0} - count : 9 - value : 1.4444444444444 - stddev : 1.8324912786484 - derivative : 1.25 - derivative_stddev : 32.234489440918 - counter : 16.666666030884 - counter_stddev : 18.966344833374 --- Adding rollup value {t=900, v=3.0} - count : 10 - value : 1.6 - stddev : 1.7999999523163 - derivative : 3.3333332538605 - derivative_stddev : 30.956954956055 - counter : 17.1428565979 - counter_stddev : 17.598121643066 </td> </tr></tbody></table> <table border="1" cellpadding="0" cellspacing="0" id="table15018" style="border-collapse:collapse; border:1px solid #afafaf;width:100%;"><tbody> <tr></tr> <tr></tr> </tbody></table><div> </div><div> </div>

6000171359

208 6000119790

2017-09-20T17:00:29-04:00

6003235491

Circonus Rollup Logic

2016-04-04T03:56:35-04:00

6003235491

2016-12-14T06:43:54-05:00

Q: How can I submit StatsD data to Circonus? Circonus offers a StatsD check that allows the collection of StatsD data on Circonus Enterprise brokers. On the Enterprise broker, incoming StatsD data is identified by the IP of the sending host and the metric name, and then sent to the corresponding check. So, make sure that the target host configured in the check matches the source address of the packets that arrive at the broker. Q: Why can't I just point the StatsD client at one of the public Circonus brokers? The reason Circonus does not accept StatsD data on public brokers is that the StatsD protocol does not allow for authentication and security. So there is no reliable way to route the incoming data to the right check. Enterprise brokers should be installed inside a LAN which is protected from illegitimate requests by a firewall. Q: How can I aggregated StatsD data from several nodes in a single metric? As mentioned before, StatsD data is routed by the source IP address (and the metric name), so data that arrives from different nodes will end up in different checks. There are two ways to work around this limitation: (a) Setup a statsD repeater. StatsD supports repeaters, which forward all incoming metrics to an upstream host. You can setup such a repeater to forward the incoming data to the Circonus broker. In this way, you have a single host IP that we can attribute that data to: https://github.com/etsy/statsd/blob/master/backends/repeater.js https://github.com/etsy/statsd/blob/master/exampleConfig.js#L88 (b) Aggregate data in a separate CAQL check. CAQL checks can be used to aggregate data across different metrics. Since percentiles can't be aggregated (Statistics for Engineers @ ACMQ), it's critical that you store your data as Histogram metrics for this method to work. An example could look as follows: search:metric:histogram("(metric:GET`/some/endpoint`timing) (tag:db)") | histogram:merge() This CAQL statement would aggregate all metrics which match the search query: "(metric:GET`/some/endpoint`timing) (tag:db)" TIP: Use the search function in the Metrics page to check that these are the metrics you are looking for.

<h2 style="">Q: How can I submit StatsD data to Circonus?</h2> Circonus offers a <a href="https://login.circonus.com/user/docs/Data/CheckTypes/statsd">StatsD check</a> that allows the collection of StatsD data on Circonus Enterprise brokers. On the Enterprise broker, incoming StatsD data is identified by the IP of the sending host and the metric name, and then sent to the corresponding check. So, make sure that the target host configured in the check matches the source address of the packets that arrive at the broker. <h2 style="">Q: Why can't I just point the StatsD client at one of the public Circonus brokers?</h2> The reason Circonus does not accept StatsD data on public brokers is that the StatsD protocol does not allow for authentication and security. So there is no reliable way to route the incoming data to the right check. Enterprise brokers should be installed inside a LAN which is protected from illegitimate requests by a firewall. <h2 style="">Q: How can I aggregated StatsD data from several nodes in a single metric?</h2> As mentioned before, StatsD data is routed by the source IP address (and the metric name), so data that arrives from different nodes will end up in different checks. There are two ways to work around this limitation: (a) Setup a statsD repeater. StatsD supports repeaters, which forward all incoming metrics to an upstream host. You can setup such a repeater to forward the incoming data to the Circonus broker. In this way, you have a single host IP that we can attribute that data to:<ul> <li style=""><a href="https://github.com/etsy/statsd/blob/master/backends/repeater.js">https://github.com/etsy/statsd/blob/master/backends/repeater.js</a></li> <li style=""><a href="https://github.com/etsy/statsd/blob/master/exampleConfig.js#L88">https://github.com/etsy/statsd/blob/master/exampleConfig.js#L88</a></li> </ul> (b) Aggregate data in a separate CAQL check. CAQL checks can be used to aggregate data across different metrics. Since percentiles can't be aggregated (<a href="http://queue.acm.org/detail.cfm?id=2903468">Statistics for Engineers @ ACMQ</a>), it's critical that you store your data as Histogram metrics for this method to work. An example could look as follows: <pre data-code-brush="text" rel="highlighter">search:metric:histogram("(metric:GET`/some/endpoint`timing) (tag:db)") | histogram:merge()</pre> This CAQL statement would aggregate all metrics which match the search query: "(metric:GET`/some/endpoint`timing) (tag:db)" TIP: Use the search function in the Metrics page to check that these are the metrics you are looking for.

6000171359

74 6000163180

2016-12-19T11:49:35-05:00

6003235491

Circonus StatsD FAQ

2016-12-14T06:43:54-05:00

6003691945

2017-08-16T11:32:17-04:00

Our new integration with Microsoft Azure allows data collection to be configured automatically through Circonus. How it Works Step 1: If you don’t have one already, create your Circonus account. Step 2: Login to your Circonus account and provide your Microsoft Azure credentials (Tenant, Client, and Secret). Simply login into your Circonus account and provide the credentials for the Azure Instances you want to monitor. The Circonus - Azure Integration will automatically go through all the resource groups for all your subscriptions and populate the Circonus Time-Series Database (powered by IRONdb) with your Microsoft Azure data. Circonus takes the credentials you provide, your Tenant, Client, and Secret, then polls the endpoint for a list of all available metrics that are specific to the user (Tenant, Client, and Secret combination). Once monitoring is enabled you will see the the names of the instances displayed. For example, say you are using Azure to run 5 web servers and 2 SQL servers. Simply provide your credentials and Circonus will automatically pull every metric associated with those servers through Azure’s data collection system and populate them into the IRONdb database in Circonus. Step 3: You’re done! Once data collection is enabled, you can then use all the graphing, analytics, and alerting tools in the Circonus platform with your Azure data. Alerting on Azure Data You can set alerts within Circonus to send notifications allowing you to make changes to the resources within Azure. For example, you can monitor the CPU usage and disk reads and writes of the web hosts you spin up in Azure, and then use this data to determine whether you should launch additional instances to handle increased load. You can also use this data to stop underused instances. With Circonus you gain system-wide visibility into resource utilization, application performance, and operational health. Support The Circonus team is here to help. Please contact sales@circonus.com for questions about our Microsoft Azure integration.

Our new integration with <a href="https://azure.microsoft.com/">Microsoft Azure</a> allows data collection to be configured automatically through Circonus. <h2 dir="ltr">How it Works</h2> Step 1: If you don’t have one already, <a href="https://login.circonus.com/signup">create your Circonus account</a>. Step 2: Login to your Circonus account and provide your Microsoft Azure credentials (Tenant, Client, and Secret). Simply login into your Circonus account and provide the credentials for the Azure Instances you want to monitor. The Circonus - Azure Integration will automatically go through all the resource groups for all your subscriptions and populate the Circonus Time-Series Database (powered by <a href="https://www.circonus.com/irondb/">IRONdb</a>) with your Microsoft Azure data. Circonus takes the credentials you provide, your Tenant, Client, and Secret, then polls the endpoint for a list of all available metrics that are specific to the user (Tenant, Client, and Secret combination). Once monitoring is enabled you will see the the names of the instances displayed. For example, say you are using Azure to run 5 web servers and 2 SQL servers. Simply provide your credentials and Circonus will automatically pull every metric associated with those servers through Azure’s data collection system and populate them into the IRONdb database in Circonus. Step 3: You’re done! Once data collection is enabled, you can then use all the graphing, analytics, and alerting tools in the Circonus platform with your Azure data. <h2 dir="ltr">Alerting on Azure Data</h2>You can set alerts within Circonus to send notifications allowing you to make changes to the resources within Azure. For example, you can monitor the CPU usage and disk reads and writes of the web hosts you spin up in Azure, and then use this data to determine whether you should launch additional instances to handle increased load. You can also use this data to stop underused instances. With Circonus you gain system-wide visibility into resource utilization, application performance, and operational health. <h2 dir="ltr">Support</h2>The Circonus team is here to help. Please contact <a href="mailto:sales@circonus.com">sales@circonus.com</a> for questions about our Microsoft Azure integration.

6000171359

97 24000013515

2017-08-16T11:33:48-04:00

6003235491

Azure Integration

2017-08-16T11:32:17-04:00

6003235491