6000110162 2015-08-21T13:01:13-04:00 6000171360 false Alerting 2 2015-10-30T11:00:31-04:00 1 1 2015-09-24T18:25:10-04:00 0 0 Initial HipChat Integration Sample While our dev team is investigating the idea of making HipChat one of the options for a contact group, we've come up with a simple way to push alerts into your HipChat rooms using our webhook feature in rules. Both of these methods involve getting a V1 API key from the HipChat Group Admin page, located here at "https://.hipchat.com/admin/api." You'll also need to know the API ID of the room you want to message, which can be found by clicking on the room in "https://.hipchat.com/admin/api." Option 1 The first option is to create a contact that posts against the HipChat API directly. This is the quickest way to send a message, but it's very limited in that you have to define the message in the URL. If you create a new contact group and add a new member, you can enter the following URL into the "New Contact" field: https://api.hipchat.com/v1/rooms/message?auth_token=ENTERAUTHTOKENHERE&room_id=ENTERROOMIDHERE&from=Circonus&message=Error+In+Circonus&format=json You can modify the message in this, but it's hard-coded into the contact, so you'd probably want to make different contacts for the different messages you want. This is limited, but is the quickest to get going. Option 2 The second option is to have an HTTP server set up that can accept POST requests from Circonus alerts, use that data, then form an API call to HipChat. An example of this created in Python using the Tornado library is located on Github. If you'd like to get this going, you'll need to modify the file and enter your auth_token and room_id, plus you can change the port. To start the server, you'll need to install Pip (if necessary), the "requests" Python library, and the "tornado" Python library. Here are the commands to do that: wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py python get-pip.py pip install requests pip install tornado python example_tornado_http_server.py Once this is running (meaning it's sitting with a blank line), you can set up a contact group with a new member set to: http://IPOFTORNADOSERVER:8080/hipchat/?format=json The Tornado server then captures the alert data, grabs the fields, figures out if it's a clear event or a new event, and then posts a detailed message to the HipChat room. Here's an example of the event details, along with the corresponding HipChat message: {u'alerts': [{u'metric_name': u'rtt', u'check_name': u'Google dns test', u'alert_value': u'9.64', u'alert_id': u'8347059', u'agent': u'Ashburn, VA, US', u'check_id': 77831, u'host': u'8.8.4.4', u'alert_time': u'Tue, 18 Feb 2014 19:19:07', u'alert_url': u'https://circonus.com/account/parlette/fault-detection?alert_id=8347059', u'metric_notes': u'Check dashboard, restart server', u'metric_link': u'', u'check_bundle_id': 57620, u'severity': u'2'}], u'account_name': u'Parlette Demo'} Circonus Alert (2:42 PM): SEVERITY 2 - Google dns test - 8.8.4.4 - rtt - Value = 9.64 -https://circonus.com/account/parlette/fault-detection?alert_id=8347059  The fields used in the message can be easily modified based on what is useful to you. <h4 style="margin-top: 10px; margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 17.5px;">Initial HipChat Integration Sample</h4> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">While our dev team is investigating the idea of making HipChat one of the options for a contact group, we've come up with a simple way to push alerts into your HipChat rooms using our webhook feature in rules. Both of these methods involve getting a V1 API key from the HipChat Group Admin page, located here at "https://.hipchat.com/admin/api." You'll also need to know the API ID of the room you want to message, which can be found by clicking on the room in "https://.hipchat.com/admin/api."</p> <h4 style="margin-top: 10px; margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 17.5px;">Option 1</h4> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">The first option is to create a contact that posts against the HipChat API directly. This is the quickest way to send a message, but it's very limited in that you have to define the message in the URL. If you create a new contact group and add a new member, you can enter the following URL into the "New Contact" field:</p> <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;"><a href="https://api.hipchat.com/v1/rooms/message?auth_token=ENTERAUTHTOKENHERE&amp;room_id=ENTERROOMIDHERE&amp;from=Circonus&amp;message=Error+In+Circonus&amp;format=json" style="color: rgb(59, 115, 185); background-color: rgb(255, 255, 255);">https://api.hipchat.com/v1/rooms/message?auth_token=ENTERAUTHTOKENHERE&amp;room_id=ENTERROOMIDHERE&amp;from=Circonus&amp;message=Error+In+Circonus&amp;format=json</a></p></blockquote> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">You can modify the message in this, but it's hard-coded into the contact, so you'd probably want to make different contacts for the different messages you want. This is limited, but is the quickest to get going.</p> <h4 style="margin-top: 10px; margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 17.5px;">Option 2</h4> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">The second option is to have an HTTP server set up that can accept POST requests from Circonus alerts, use that data, then form an API call to HipChat. An example of this created in Python using the Tornado library is located on <a href="https://github.com/circonus-labs/circonus-webhooks-python" style="color: rgb(59, 115, 185);">Github</a>. If you'd like to get this going, you'll need to modify the file and enter your auth_token and room_id, plus you can change the port. To start the server, you'll need to install Pip (if necessary), the "requests" Python library, and the "tornado" Python library. Here are the commands to do that:</p> <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">wget <a href="https://raw.github.com/pypa/pip/master/contrib/get-pip.py" style="color: rgb(59, 115, 185);">https://raw.github.com/pypa/pip/master/contrib/get-pip.py</a> python get-pip.py pip install requests pip install tornado python example_tornado_http_server.py</p></blockquote> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">Once this is running (meaning it's sitting with a blank line), you can set up a contact group with a new member set to:</p> <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;"><a href="http://ipoftornadoserver:8080/hipchat/?format=json" style="color: rgb(59, 115, 185);">http://IPOFTORNADOSERVER:8080/hipchat/?format=json</a></p></blockquote> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">The Tornado server then captures the alert data, grabs the fields, figures out if it's a clear event or a new event, and then posts a detailed message to the HipChat room. Here's an example of the event details, along with the corresponding HipChat message:</p> <blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">{u'alerts': [{u'metric_name': u'rtt', u'check_name': u'Google dns test', u'alert_value': u'9.64', u'alert_id': u'8347059', u'agent': u'Ashburn, VA, US', u'check_id': 77831, u'host': u'8.8.4.4', u'alert_time': u'Tue, 18 Feb 2014 19:19:07', u'alert_url': u'<a href="https://circonus.com/account/parlette/fault-detection?alert_id=8347059%27," style="color: rgb(59, 115, 185);">https://circonus.com/account/parlette/fault-detection?alert_id=8347059',</a> u'metric_notes': u'Check dashboard, restart server', u'metric_link': u'', u'check_bundle_id': 57620, u'severity': u'2'}], u'account_name': u'Parlette Demo'}</p> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">Circonus Alert (2:42 PM): SEVERITY 2 - Google dns test - 8.8.4.4 - rtt - Value = 9.64 -<a href="https://circonus.com/account/parlette/fault-detection?alert_id=8347059" style="color: rgb(59, 115, 185);">https://circonus.com/account/parlette/fault-detection?alert_id=8347059</a> </p> </blockquote> <p style="margin-bottom: 10px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">The fields used in the message can be easily modified based on what is useful to you.</p> 6000171360 127 6000066953 2015-10-23T12:04:45-04:00 6003235491 1 2 0 0 Hip Chat and Alerts 2015-09-24T18:25:10-04:00 6003235491 1 2015-09-24T18:26:36-04:00 0 0 How can we control an alert that goes into an alarm state and sends 1000s of emails? For example, a group of servers encounters an issue due to a common event and until the issue is resolved constantly flaps between alert and recovery, sending 1000s of emails. There are 3 ways to handle this: If these all depend on some other issue, you could set the dependencies up so that if the common event happens, the dependent items don't alert until that sev 1 is cleared.  Aggregation windows on the contact group will queue up emails and limit the number that are sent out.  If the alert is going back and forth between alerting and clearing with each polling cycle, you can set your alerting rule to only trigger if the threshold has been breached for multiple polling cycles, such as 3-5 minutes for the standard 1-minute check. It is also possible that you could solve this with ruleset groups and expressions. If you'd like more details about how to accomplish this, contact support@circonus.com -cmadsen <div dir="ltr"> <p><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">How can we control an alert that goes into an alarm state and sends 1000s of emails? For example, a group of servers encounters an issue due to a common event and until the issue is resolved constantly flaps between alert and recovery, sending 1000s of emails.</span></p> <p><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;"><br></span></p> <p><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">There are 3 ways to handle this: </span></p> <p><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;"> </span></p> <p></p> <ol> <li><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">If these all depend on some other issue, you could set the dependencies up so that if the common event happens, the dependent items don't alert until that sev 1 is cleared. </span></li> <li><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">Aggregation windows on the contact group will queue up emails and limit the number that are sent out. </span></li> <li> <span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">If the alert is going back and forth between alerting and clearing with each polling cycle, you can set your alerting rule to only trigger if the threshold has been breached for multiple polling cycles, such as 3-5 minutes for the standard 1-minute check.</span> </li> </ol> <p><span style="font-size: 14px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;"><br></span></p> <p><span style="font-size: 14px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;">It is al</span><span style="font-size: 14px; font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;">so possible that you could solve this with ruleset groups and expressions. If you'd like more details about how to accomplish this, contact </span><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">support@circonus.com </span></p> <p><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;"><br></span></p> <p><span style="font-family: 'Open Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px;">-cmadsen</span></p> </div> 6000171360 232 6000066954 2016-11-15T11:31:29-05:00 6003235491 2 2 0 0 Alert Flapping - How to handle flapping alerts? 2015-09-24T18:26:36-04:00 6003235491 1 2016-02-17T10:16:46-05:00 0 0 Alerting is key to any monitoring system. Collecting data allows you to perform historic analysis and identify the root cause of a problem after the fact, but to ensure the best possible customer experience and meet "always on" expectations, IT professionals need to be able to respond to problems in their environment as soon as possible. They need a monitoring system with the flexible alerting tools that best enable them to act on incoming data. That's why Circonus is expanding on its existing alerting capabilities. Users will already be familiar with threshold-based alerts in Circonus, but now Circonus has a new Analytics Alerting feature that enables you to create alerts based on CAQL transformed metrics. This opens up a wide range of advanced application scenarios for alerting. In this a tutorial, we'll discuss just a few of these use case scenarios; SOA Monitoring, Velocity Alerting, and Anomaly Alerting. SOA Monitoring with Histograms and Percentiles Percentile Overlays are commonly used in Service Oriented Architecture (SOA) Monitoring in order to validate Service Level Agreement (SLA) requirements. Historical analysis of this data allows IT to identify the causes of performance spikes, and allows your company to plan ahead and create practical SLAs based on actual historical performance instead of arbitrary values. However, the ability to alert on this data in real-time is tremendously important for SOA Monitoring, because it allows IT to take rapid action to avoid or correct SLA violations as they occur. Let's look at a practical example of how you could use Analytics Alerting to monitor API performance: First, we create or open a graph containing an API latency histogram metric. Next we select an appropriate percentile that reflects our performance expectations for the API. We can use percentile overlays to inspect the behavior of different percentiles over historic time windows. For this example, we'll assume this metric must stay below the 80th percentile.                                                              Figure 1: API Latency Histogram Now we need to navigate to the rules page for that particular metric. The quickest way to do is to hover the information symbol (i) for the metric in the graph legend and click the "view" link for the metric. This will open a metric overview page displaying a list with a single item: this metric.                                    Figure 2: Graph Legend information button. Note the "view" metric link. On the metric overview page, click the menu button for the metric, and then the "Set Rules" link. This will display the Rulesets page.                                                                   Figure 3: Metric menu. Note the Set Rules link. On the Rulesets page for the metric, create the rule using the "Add Analytics Rule" button.                         Figure 4: New Rulesets page. Note the new "Add Analytics Rule +" button. In this example, we will create a rule to alert when our API performance metrics rise above the 80th percentile. Select "Percentile Alerting" from the drop down and set the field to 80.                                          Figure 5: "Add an Analytics Rule" dialog By saving the rule, we activate alerting and will receive alerts if and whenever the rule is violated in the future. Read more about the applications of histograms in API monitoring in the ACM Queue article, Statistics for Engineers, by our Chief Data Scientist, Heinrich Hartmann. Werner Vogels, Amazon CTO, describes the importance of this kind of monitoring in his paper, Amazon's Dynamo, paragraph 2.1 Velocity Alerting and Forecasting Forecasting future data based on current trends is the holy grail of statistical analysis. The existing Capacity Planning overlays make predictions based on current data to enable your company to plan ahead for the growing needs of your IT environment. How soon will you need more disk space? When will you run out of memory? These are Capacity Planning questions. Velocity Alerting is creating alerts based on forecasted data, so you can be notified about what you need before you run out. Without alerting, you would need to check your forecasts regularly to plan ahead, but now you can set up a ruleset and tell Circonus, just for example: "Alert me when I will be running out of disk space in 2 weeks." "Alert me when I will be out of memory in 2 hours." "Tell me if my disk usage will reach 1 Tb in less than 5 days." "Tell me when my CPU load, across my Metric Cluster, will get to 80%." Let's look at another practical example of how to set up Smart Alerting to create a Velocity Alert. For this example, we assume that we are monitoring resource usage (for example, disk space) with a numeric metric: We can examine forecasts of future values with a Capacity Planning overlay.                                           Figure 6: Resource Usage with Capacity Planning overlay We navigate to the "Ruleset" page for this particular metric, as described in the previous section. We add an Analytics rule. Click the "Add Analytics Rule +" button.                                          Figure 7: "Forecasted Value" options Select "Velocity Alerting" from the drop down and select the following: The amount of time we want to forecast into the future The threshold value on which we wants to alert Optionally, suggest the precise forecasting method that should be used, or leave this unspecified and let Circonus select the model that best fits the data. We save the rule in order to start receiving alerts. For more information about the forecasting functions, check out our article on Forecasting Values with CAQL. Anomaly Alerting Anomaly Detection Overlays highlight areas where collected data doesn't conform to expectations. It can be a huge time saver when conducting historical analysis to identify when things started to go wrong. Now you can receiver alerts in real-time when there are anomalies in your data. This allows IT rapidly respond to unexpected changes in your data stream, which could be signs of trouble, without necessarily specifying what those changes would be ahead of time. Instead of alerting on a specific threshold, you can choose to alert just because the data deviates too much from what you expect. Warning: Anomaly Detection has a high probability of generating false positives, especially in real-time data. You can mitigate this by lowering the sensitivity of your Anomaly Detection Overlays, but there is the potential to be flooded with unwarranted alerts. Here's an example of how you could set up Anomaly Alerting. Let's assume that we're monitoring ping/dns latencies or request rates using a numeric metric. We examine anomalies using a graph overlay. We tune the sensitivity parameter until only events that we cares about deeply are highlighted by the Anomaly Detection Overlay in the graph. These should be the sort of events we'd want to be alerted about in the future.                                                   Figure 8: Anomaly Detection Overlay We navigate to the "Ruleset" page for this particular metric, as described above. Click the "Add Analytics Rule +" button. We select "Anomaly Alerting" from the drop down menu.                                          Figure 9: Anomaly Detection rule options Now we can review the rule and change the model or adjust the sensitivity parameter. Be careful, because a very high sensitivity can result in false positives that will spam your alerts.                                          Figure 10: Adjusting the sensitivity parameter We save the rule so we can begin receiving alerts. Be aware that anomalies are not the same faults, so you can expect anomaly alerts that do not point to failure cases. You can tune the sensitivity to help minimize those false positives. For more information about Anomaly Detection in Circonus, refer to our user documentation. More Information Analytics Alerting with CAQL is a very flexible tool. In addition to the above examples, you can also alert on CAQL statements, which exposes additional parameters and options. The "Convert to CAQL" button allows you to convert any other type of Analytics Alert into a CAQL alert, or you can write your own CAQL statements from scratch. For more information about CAQL and it's many functions, refer to the CAQL Reference Manual, or check out some of the articles about CAQL here on the Support Portal. Disclaimer: The Beaker component that runs these alerts is not highly available, right now. Some downtime is to be expected in the next few months. Don't depend too much on the reliability of this feature, until the Beaker is out of Beta. Check the step-by-step instructions in our User Manual to learn more about Alerting with Analytics. <p style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Alerting is key to any monitoring system. Collecting data allows you to perform historic analysis and identify the root cause of a problem after the fact, but to ensure the best possible customer experience and meet "always on" expectations, IT professionals need to be able to respond to problems in their environment as soon as possible. They need a monitoring system with the flexible alerting tools that best enable them to act on incoming data. That's why Circonus is expanding on its existing alerting capabilities.</p><p style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><br></p><p style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Users will already be familiar with <a href="6000040841-create-an-alert" target="">threshold-based alerts</a> in Circonus, but now Circonus has a new Analytics Alerting feature that enables you to create alerts based on <a href="6000055846-caql-v1-0-tutorial" target="">CAQL</a> transformed metrics. This opens up a wide range of advanced application scenarios for alerting. In this a tutorial, we'll discuss just a few of these use case scenarios; SOA Monitoring, Velocity Alerting, and Anomaly Alerting.</p><p style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><br></p><h2>SOA Monitoring with Histograms and Percentiles</h2><p>Percentile Overlays are commonly used in <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Service Oriented Architecture (SOA) Monitoring in order to validate</span><span> Service Level Agreement (SLA) requirements. Historical analysis of this data allows IT to identify the causes of performance spikes, and allows your company to plan ahead and create practical SLAs based on actual historical performance instead of arbitrary values. However, the ability to alert on this data in real-time is tremendously important for </span><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">SOA</span><span> Monitoring, because it allows IT to take rapid action to avoid or correct SLA violations as they occur.</span></p><p><br></p><p>Let's look at a practical example of how you could use Analytics Alerting to monitor API performance:</p><ol> <li> <span>First, we c</span><span>reate or open a graph containing an API latency histogram metric</span><span>.<br><br></span> </li> <li>Next we select an appropriate percentile that reflects our performance expectations for the API. We can use percentile overlays to inspect the behavior of different percentiles over historic time windows. For this example, we'll assume this metric must stay below the 80th percentile.<br><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021212727/original/blob1461684435270.png?1461684434" class="inline-image fr-dii fr-draggable" data-id="6021212727" data-height="317" style="height: 317px;"></p> <strong><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif; font-weight: normal;">                                                             </span>Figure 1:</strong> API Latency Histogram<br><br> </li> <li>Now we need to navigate to the rules page for that particular metric. The quickest way to do is to hover the information symbol (i) for the metric in the graph legend<span> and click the "view" link for the metric. This will open a metric overview page displaying a list with a single item: this metric.</span><br><br><p>                       <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021047970/original/blob1461548234681.png?1461548234" class="inline-image fr-dii fr-draggable" data-id="6021047970"></p> <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">           </span><b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Figure 2:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> Graph Legend information button. Note the "view" metric link.<br></span><br> </li> <li> <span>On the metric overview page, click the menu button for the metric, and then<span> the "Set Rules" link.</span> This will display the Rulesets page.<br><br></span><span><p style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">                                         <img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021048056/original/blob1461548649494.png?1461548649" data-id="6021048056" style="font-size: 13px;" class="fr-dii fr-draggable"><span><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><span style="font-size: 13px; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><br>                      </span>  </span><b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Figure 3:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> Metric menu. Note the Set Rules link.</span><br><br></span></p></span> </li> <li><span><span>On the Rulesets page for the metric, create the rule using the "Add Analytics Rule" button.</span><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021048463/original/blob1461550211900.png?1461550212" class="inline-image fr-dii fr-draggable" data-id="6021048463" data-height="332" style="height: 332px;"></p> <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><span style="font-size: 13px; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">                      </span>  </span><b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Figure 4:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> New Rulesets page. Note the new "Add Analytics Rule +" button.</span><br><br></span></li> <li><span>In this example, we will create a rule to alert when our API performance metrics rise above the 80th percentile. Select "Percentile Alerting" from the drop down and set the field to 80.<br><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021107020/original/blob1461597749352.png?1461597748" class="inline-image fr-dii fr-draggable" data-id="6021107020" data-height="213" style="height: 213px;"></p> <b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif; font-weight: normal;">                                         </span>Figure 5:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> "Add an Analytics Rule" dialog<br><br></span></span></li> <li> <span>By saving the rule, we activate alerting and will receive alerts if and whenever the rule is violated in the future.</span><br> </li> </ol><p><br></p><p>Read more about the applications of histograms in API monitoring in the <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">ACM Queue </span><span>article, </span><a href="http://queue.acm.org/detail.cfm?id=2903468" style="background-color: rgb(255, 255, 255);" target="">Statistics for Engineers</a><span>, by our Chief Data Scientist, Heinrich Hartmann. Werner Vogels, Amazon CTO, describes the importance of this kind of monitoring in his paper, </span><a href="http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html" style="background-color: rgb(255, 255, 255);" target="">Amazon's Dynamo</a>,<span> paragraph 2.1</span></p><p><br></p><h2>Velocity Alerting and Forecasting</h2><p>Forecasting future data based on current trends is the holy grail of statistical analysis. The existing Capacity Planning overlays make predictions based on current data to enable your company to plan ahead for the growing needs of your IT environment. How soon will you need more disk space? When will you run out of memory? These are Capacity Planning questions.</p><p><br></p><p>Velocity Alerting is creating alerts based on forecasted data, so you can be notified about what you need before you run out. <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Without alerting, you would need to check your forecasts regularly to plan ahead, but now you can set up a ruleset and tell Circonus, just for example:</span></p><ul> <li> <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">"</span><span>Alert me when I will be running out of disk space </span><span data-term="goog_1737923161">in 2 weeks."</span> </li> <li> <span data-term="goog_1737923161">"</span><span>Alert me when I will be out of memory in 2 hours."</span> </li> <li>"Tell me if my disk usage will reach 1 Tb in less than 5 days."</li> <li>"Tell me when my CPU load, across my Metric Cluster, will get to 80%."</li> </ul><p><br></p><p>Let's look at another practical example of how to set up Smart Alerting to create a Velocity Alert. For this example, we assume that we are <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">monitoring resource usage (for example, disk space) with a numeric metric:<br><br></span></p><ol> <li><span>We can examine forecasts of future values with a Capacity Planning overlay.<br><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021208985/original/blob1461682494665.png?1461682493" class="inline-image fr-dii fr-draggable" data-id="6021208985" data-height="267" style="height: 267px;"></p> <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">                                         </span> </span><b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Figure 6:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> Resource Usage with Capacity Planning overlay<br></span><br></span></li> <li> <span>We navigate to the "Ruleset" page for this particular metric, as described in the previous section.<br></span><br> </li> <li><span>We add an Analytics rule. <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Click the "Add Analytics Rule +" button.<br><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021048507/original/blob1461550334648.png?1461550334" class="inline-image fr-dii fr-draggable" data-id="6021048507" data-height="229" style="height: 229px; width: 630.938px;"></p> <b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif; font-weight: normal;">                                         </span>Figure 7:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> "Forecasted Value" options</span><br><br></span></span></li> <li> <span><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Select "Velocity Alerting" from the drop down and select the following:</span></span><ol> <li><span>The amount of time we want to forecast into the future</span></li> <li><span>The threshold value on which we wants to alert</span></li> <li><span>Optionally, suggest the precise forecasting method that should be used, or leave this unspecified and let Circonus select the model that best fits the data.<br><br></span></li> </ol> </li> <li><span>We save the rule in order to start receiving alerts.</span></li> </ol><p><span style="line-height: 18px;"></span></p><p><br></p><p>For more information about the forecasting functions, check out our article on <a href="6000116508-forecasting-values-with-caql" target="">Forecasting Values with CAQL</a>.</p><p><br></p><h2>Anomaly Alerting</h2><p>Anomaly Detection Overlays highlight areas where collected data doesn't conform to expectations. It can be a huge time saver when conducting historical analysis to identify when things started to go wrong. Now you can receiver alerts in real-time when there are anomalies in your data. This allows IT rapidly respond to unexpected changes in your data stream, which could be signs of trouble, without necessarily specifying what those changes would be ahead of time. Instead of alerting on a specific threshold, you can choose to alert just because the data deviates too much from what you expect.</p><p><br></p><p><strong>Warning:</strong> Anomaly Detection has a high probability of generating false positives, especially in real-time data. You can mitigate this by lowering the sensitivity of your Anomaly Detection Overlays, but there is the potential to be flooded with unwarranted alerts.<br></p><p><br></p><p>Here's an example of how you could set up Anomaly Alerting. Let's assume that we're <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">monitoring ping/dns latencies or request rates using a numeric metric.</span></p><ol> <li><span>We examine anomalies using a <a href="https://login.circonus.com/user/docs/Visualization/Graphs/View/Overlays/Analytics" target="">graph overlay</a>.<br><br></span></li> <li> <span>We tune the sensitivity parameter until only events that we cares about deeply are highlighted by the Anomaly Detection Overlay in the graph. These should be the sort of events we'd want to be alerted about in the future.<br></span><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021209054/original/blob1461682526752.png?1461682525" class="inline-image fr-dii fr-draggable" data-id="6021209054" data-height="227" style="height: 227px;"></p> <b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif; font-weight: normal;">                                                  </span>Figure 8:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> Anomaly Detection Overlay<br></span><br> </li> <li> <span>We navigate to the "Ruleset" page for this particular metric, as described above.<br></span><br> </li> <li><span><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Click the "Add Analytics Rule +" button.<br></span><br></span></li> <li> <span>We select "Anomaly Alerting" from the drop down menu.<br></span><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021047284/original/blob1461545053266.png?1461545053" class="inline-image fr-dii fr-draggable" data-id="6021047284" data-height="213" style="width: 631.012px; height: 213px;"></p> <span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">                                         </span><b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;">Figure 9:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> Anomaly Detection rule options<br></span><br> </li> <li> <span>Now we can review the rule and change the model or adjust the sensitivity parameter. Be careful, because a very high sensitivity can result in false positives that will spam your alerts.<br><p><img src="https://s3.amazonaws.com/cdn.freshdesk.com/data/helpdesk/attachments/production/6021047490/original/blob1461546032632.png?1461546032" class="inline-image fr-dii fr-draggable" data-id="6021047490" data-height="122" style="height: 122px;"></p></span><b style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"><span style="font-weight: normal;">                                         </span>Figure 10:</b><span style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif, 'Helvetica Neue', Helvetica, Arial, sans-serif;"> Adjusting the sensitivity parameter<br></span><br> </li> <li> <span>We save the rule so we can begin receiving alerts.</span><br> </li> </ol><p><br></p><p>Be aware that anomalies are not the same faults, so you can expect anomaly alerts that do not point to failure cases. You can tune the sensitivity to help minimize those false positives.</p><p><br></p><p>For more information about Anomaly Detection in Circonus, refer to our <a href="https://login.circonus.com/user/docs/Visualization/Graphs/View/Overlays/Analytics#AnomalyDetection" target="">user documentation</a>.</p><p><br></p><h2>More Information</h2><p><span>Analytics Alerting with CAQL is a very flexible tool. In addition to the above examples, you can also alert on CAQL statements, which exposes additional parameters and options. The "Convert to CAQL" button allows you to convert any other type of Analytics Alert into a CAQL alert, or you can write your own CAQL statements from scratch. For more information about CAQL and it's many functions, refer to the </span><a href="https://login.circonus.com/user/docs/caql_reference" target="">CAQL Reference Manual</a><span>, or check out some of the </span><a href="/solution/folders/6000191056" target="">articles about CAQL</a><span> here on the Support Portal.</span><br></p><p><br></p><div> <strong>Disclaimer:</strong> The Beaker component that runs these alerts is not highly available, right now. Some downtime is to be expected <span>in the next few months. Don't depend too much on the reliability of this feature, until the Beaker is out of Beta.</span><p><br></p> <p>Check the step-by-step instructions in our <a href="https://login.circonus.com/user/docs/Alerting/Transforms" style="background-color: rgb(255, 255, 255);" target="">User Manual</a> to learn more about Alerting with Analytics.</p> </div> 6000171360 420 6000112370 2017-09-20T17:11:37-04:00 6003235491 3 2 0 0 Analytics Alerting Tutorial and Use Cases 2016-04-26T18:28:38-04:00 6003235491