我的網誌清單

2023年2月13日星期一

Writing blogs

發佈者： 於 沒有留言:

Trading Bots

Market data into spreadsheet

https://www.alphavantage.co/spreadsheets/.    data can be shown with flask



GitHub - birdhouses/ai-trader     python + open AI

GitHub - birdhouses/trading212: Get insights in your trading212 dividend payouts




Copilot answers APIdog to scrap the data from Tradingview

To use Apidog for generating a schema from TradingView's screener database, you would need to follow these steps. However, note that scraping TradingView's data directly may violate their terms of service, so it's always better to use their official API if available. Here's a general guide for schema generation using Apidog:

1. Understand the Data Source

  • TradingView does not provide direct access to its database schema. If you're trying to scrape or analyze screener data, you might need to first identify the structure of the data (e.g., JSON responses from their API or web requests).
  • Use tools like browser developer tools (Network tab) to inspect the data format (e.g., JSON or HTML).

2. Use Apidog to Generate Schema

Apidog supports schema generation from JSON or database tables. Here's how you can proceed:

Option 1: From JSON Data

  1. Extract JSON Data:

    • If you have access to TradingView's screener data in JSON format (e.g., from API responses or web scraping), save it as a .json file.

  2. Generate Schema:

    • Open Apidog and navigate to the "Generate Schema" section.
    • Upload the JSON file or paste the JSON content directly.
    • Apidog will analyze the structure and generate a schema automatically.

Option 2: From Database Tables

  1. Connect to Database:

    • If you have access to a database containing TradingView-like screener data, connect Apidog to the database.
    • Provide the necessary credentials (host, port, username, password).

  2. Generate Schema:

    • Apidog will read the table structures and generate schemas for each table.

3. Validate and Customize the Schema

  • Review the generated schema to ensure it matches the data structure.
  • Customize field names, data types, or relationships as needed.

Important Notes

  • Respect Terms of Service: Scraping or unauthorized access to TradingView's data may breach their terms of service. Always prefer using their official API if available.
  • Alternative Tools: If Apidog doesn't meet your needs, consider other schema generation tools like Postman or Swagger.

Let me know if you'd like further clarification! 😊


Python Library "TradingView_screener"

https://shner-elmo.github.io/TradingView-Screener/3.0.0/tradingview_screener.html#quickstart

GitHub:

https://github.com/shner-elmo/TradingView-Screener


It is handy to have an online SQL viewer to look at the contents of the database files.

https://beta.sqliteviewer.app/2025-08-17.db/table/stock_data


Copilot suggested python codes to write a data file to Google sheet.

Here’s a concise guide to storing a file's data into a Google Sheet using Python. This involves enabling the Google Sheets API, authenticating, and writing data to the sheet.

Steps Overview

  1. Enable the Google Sheets API in your Google Cloud Console.
  2. Download the credentials.json file for authentication.
  3. Install required Python libraries: google-auth, google-auth-oauthlib, google-auth-httplib2, and gspread.
  4. Use the following Python code to upload data.

Code Example 1: Writing Data to Google Sheets

import gspread
from google.oauth2.service_account import Credentials

# Authenticate using the credentials.json file
scope = ["https://spreadsheets.google.com/feeds", "https://www.googleapis.com/auth/drive"]
creds = Credentials.from_service_account_file("credentials.json", scopes=scope)
client = gspread.authorize(creds)

# Open the Google Sheet by name
sheet = client.open("Your Google Sheet Name").sheet1

# Example: Writing data to the sheet
data = [["Name", "Age", "City"], ["Alice", 30, "San Francisco"], ["Bob", 25, "New York"]]
for row in data:
    sheet.append_row(row)

print("Data successfully written to Google Sheet!")

Code Example 2: Reading a File and Storing Its Data

import csv
import gspread
from google.oauth2.service_account import Credentials

# Authenticate using the credentials.json file
scope = ["https://spreadsheets.google.com/feeds", "https://www.googleapis.com/auth/drive"]
creds = Credentials.from_service_account_file("credentials.json", scopes=scope)
client = gspread.authorize(creds)

# Open the Google Sheet by name
sheet = client.open("Your Google Sheet Name").sheet1

# Read data from a CSV file and write to the sheet
with open("your_file.csv", "r") as file:
    reader = csv.reader(file)
    for row in reader:
        sheet.append_row(row)

print("File data successfully written to Google Sheet!")

Code Example 3: Writing Data from a Python Dictionary

import gspread
from google.oauth2.service_account import Credentials

# Authenticate using the credentials.json file
scope = ["https://spreadsheets.google.com/feeds", "https://www.googleapis.com/auth/drive"]
creds = Credentials.from_service_account_file("credentials.json", scopes=scope)
client = gspread.authorize(creds)

# Open the Google Sheet by name
sheet = client.open("Your Google Sheet Name").sheet1

# Example: Writing dictionary data to the sheet
data = [{"Name": "Alice", "Age": 30, "City": "San Francisco"},
        {"Name": "Bob", "Age": 25, "City": "New York"}]

# Write headers
sheet.append_row(list(data[0].keys()))

# Write rows
for entry in data:
    sheet.append_row(list(entry.values()))

print("Dictionary data successfully written to Google Sheet!")

Key Notes

  • Replace "Your Google Sheet Name" with the actual name of your Google Sheet.
  • Ensure the credentials.json file is downloaded from your Google Cloud Console and placed in the same directory as your script.
  • Install required libraries using:
    pip install gspread google-auth google-auth-oauthlib google-auth-httplib2

Let me know if you need further clarification or help! 😊


Indian guy stockanalysis 

MCP codes only for India market 

https://medium.com/@varungangu1/building-a-stock-analysis-mcp-server-with-docker-and-claude-desktop-eae4963dc3a7

his GitHub depository , not much description over there

https://github.com/varun-g-12/StockAnalysisMCP


Python Library "TradingView_scrapper" live example

Saved 3 files:-  52wks high ,  all time highs, and high gainers in excel formats everyday.

then analyze the patterns, next things I would do is to save the files into my Google sheets.


data scrapped from this TradingView page:

https://www.tradingview.com/markets/stocks-usa/market-movers-52wk-high/


Original

 https://medium.com/@alvinmarco8/otomasi-scrapping-data-market-saham-di-tradingview-menggunakan-python-c499a244db40

my codes:

from urllib.request import urlopen

from bs4 import BeautifulSoup

import pandas as pd

from datetime import date

import os

from urllib.error import HTTPError

def tradingviewScrapper():


    urls = [

        'https://www.tradingview.com/markets/stocks-usa/market-movers-52wk-high/',

        'https://www.tradingview.com/markets/stocks-usa/market-movers-ath/',

        'https://www.tradingview.com/markets/stocks-usa/market-movers-gainers/'

    ]

    root_path = r'/Users/dl/Documents/codes/tradings/'

    print(f"Saving files to: {root_path}")

    os.makedirs(root_path, exist_ok=True)

    for url in urls:

        try:

            # Content retrieval

            page = urlopen(url)

            html = page.read().decode("utf-8")

            # Create BeautifulSoup object

            soup = BeautifulSoup(html, "html.parser")

            rows = soup.find_all('tr')

            parts = url.split('market-movers-')

            typeurl = parts[1]

            today = date.today().strftime('%y-%m-%d')

            data = []

            # Loop through each line and get data from element <td>

            for row in rows:

                # Get all <td> elements in a line

                cells = row.find_all('td')

                # Get the text content from each cell and add it to the data list

                row_data = [cell.text.strip() for cell in cells]

                data.append(row_data)

            th_elements = rows[0].find_all('th')

            headers = [th.text.strip() for th in th_elements]

            # Create DataFrame from data list

            df = pd.DataFrame(data, columns=headers)

            filename = f'Data_{str(typeurl)}_{str(today)}.xlsx'

            excel_path = os.path.join(root_path, filename)

            print(f"Saving Excel to: {excel_path}")          

            # Ensure the parent directory exists

            os.makedirs(os.path.dirname(excel_path), exist_ok=True)

            df.to_excel(excel_path, index=False)

            print(f'Data {url} successfully discraping.')

        except Exception as e:

            print(f"Error saving file: {e}")

tradingviewScrapper()


----------------------------------------------------------------------------------------------------------------------------

TradeBot: Stock Trading using Reinforcement Learning — Part1 | by Shivam Akhauri | ETHER Labs | Medium

發佈者： 於 沒有留言:

Home Assistant - Home Automation

發佈者： 於 沒有留言:

ESP32/ESP8266

  •  Arduino library



Flash Any ESP32 Version with the ESP32-CAM-MB USB programmerESP32 CAM MB and ESP32-Cam – ESP32 & ESP8266 – DroneBot Workshop Forums

SD Card Pin Mapping on ESP32-CAM

The ESP32-CAM uses specific GPIO pins for interfacing with the SD card. These pins are shared with other peripherals, limiting their availability for general use. The SD card connections are as follows:

SD Card Signal ESP32-CAM GPIO
HS2_CLK GPIO14.        SCLK(HSPI)
HS2_CMD GPIO15.        CS 
HS2_DATA0 GPIO2
HS2_DATA1 GPIO4 (also connected to the flash LED)
HS2_DATA2 GPIO12.      MISO (HSPI)
HS2_DATA3 GPIO13.      MOSI (HSPI)

Key Notes:

  • GPIO4 is connected to the onboard flash LED, which may cause it to flash during SD card operations.

  • GPIO12 must be LOW during boot to avoid issues with the ESP32's boot process.


    SD Card Modes and Available GPIOs

    The SD card can operate in two modes: 4-bit mode and 1-bit mode. Each mode affects the availability of GPIOs.

    4-Bit Mode

    In 4-bit mode, all six GPIOs (2, 4, 12, 13, 14, 15) are used for SD card communication. This leaves only GPIO16 available for other purposes.

    1-Bit Mode

    In 1-bit mode, the SD card uses fewer pins, freeing up GPIO12 and GPIO13 for other uses. The configuration for 1-bit mode is as follows:


    #include "SD_MMC.h"

    void setup() {
    SD_MMC.begin("/sdcard", true);  // Enable 1-bit mode
    }

    This mode reduces data transfer speed but provides additional GPIOs for peripherals.


Managing GPIO4 (Flash LED)

To prevent the flash LED from interfering with SD card operations, you can initialize it manually:

#include "SD_MMC.h"

void setup() {
pinMode(GPIO_NUM_4, OUTPUT);
digitalWrite(GPIO_NUM_4, LOW);  // Turn off the flash LED
SD_MMC.begin("/sdcard", true);  // Initialize SD card in 1-bit mode
}

Caution: Avoid keeping the flash LED on for extended periods, as it lacks a current-limiting resistor and may overheat.

Additional Tips for GPIO Management

  • GPIO16 is always available but may have limitations depending on your board's configuration.

  • If more GPIOs are needed, consider using an I2C GPIO expander like the MCP23017. This allows you to add multiple GPIOs while using only two pins (SDA and SCL).

Example of Using MCP23017

#include "Wire.h"
#include "Adafruit_MCP23X17.h"
#include "SD_MMC.h"

Adafruit_MCP23X17 mcp;

void setup() {
SD_MMC.begin("/sdcard", true);  // 1-bit mode
Wire.begin(13, 12);  // SDA=13, SCL=12

if (!mcp.begin_I2C()) {
Serial.println("MCP23017 not found!");
while (1);
}

mcp.pinMode(0, OUTPUT);  // Set pin 0 on MCP23017 as output
}

This approach can significantly expand the number of GPIOs available for your project.

Conclusion

The ESP32-CAM's GPIOs are heavily utilized by the camera and SD card. By switching to 1-bit mode, you can free up GPIO12 and GPIO13 for additional functionality. For more extensive GPIO requirements, consider using an I2C GPIO expander. These strategies allow you to maximize the ESP32-CAM's capabilities while maintaining SD card functionality.


SD gpio pins usage:-


https://stackoverflow.com/questions/73178340/esp32-cam-sd-and-camera-use-up-all-the-pins


more precise desc about sd 1 bit mode:

https://dr-mntn.net/2021/02/using-the-sd-card-in-1-bit-mode-on-the-esp32-cam-from-ai-thinker


https://github.com/alanesq/esp32cam-demo.     



Note about GPIO2 (ESP32 only)

GPIO2 pin is used as a bootstrapping pin, and should be low to enter UART download mode. One way to do this is to connect GPIO0 and GPIO2 using a jumper, and then the auto-reset circuit on most development boards will pull GPIO2 low along with GPIO0, when entering download mode.

  • Some boards have pulldown and/or LED on GPIO2. LED is usually ok, but pulldown will interfere with D0 signals and must be removed. Check the schematic of your development board for anything connected to GPIO2.



spi flash


esp32 cam spi pins

SPI Pins

The ESP32-CAM features only one SPI (VSPI) in slave and master modes. It also supports the general-purpose SPI features listed below:

  • 4 timing modes of the SPI format transfer
  • Up to 80 MHz and the divided clocks of 80 MHz
  • Up to 64-Byte FIFO
https://lastminuteengineers.com/esp32-cam-pinout-reference/.   a detailed description of esp32cam pins



https://microcontrollerslab.com/esp32-cam-ai-thinker-pinout-gpio-pins-features-how-to-program/#google_vignette.        several articles about the camera


https://gitmemories.com/index.php/jameszah/ESP32-CAM-RocketCam



hx-esp32-cam-fpv.    spectacular air video

https://github.com/rolly46/hx-esp32-cam-fpv



ftp server



https://github.com/mtnbkr88/ESP32_FTPServer.   for esp32 cam


https://github.com/nopnop2002/esp-idf-ftpServer


https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo/blob/master/MicroPython_BUILD/components/micropython/esp32/libs/ftp.c


https://github.com/espressif/esp-idf.    building environment and development software. python based


installation procedures:-


mkdir esp, cd esp,

git clone https://github.com/espressif/esp-idf

cd esp-idf

git submodule update --init --recursive

./install.sh

./export.sh

idf.py build or

idf.py main.py or

idf.py menuconfig


upload flash


idf.py -p /dev/ttyUSB0 -b 115200 flash


instructions to run build availabel


https://www.xda-developers.com/how-use-ftp-mac/.     ftp replaced with smb or sftp



SD CARD 


https://github.com/s60sc/ESP32-CAM_MJPEG2SD.    pretty new project


https://forum.arduino.cc/t/esp32s-can-t-detect-mount-microsd-card-fix/914266.  FIX


https://github.com/espressif/arduino-esp32/issues/4680




// Libraries for SD card

#include "FS.h"

#include <SD.h>

//#include "mySD.h"

#include <SPI.h>



// Define CS pin for the SD card module

#define SD_MISO     2

#define SD_MOSI     15

#define SD_SCLK     14

#define SD_CS       13

SPIClass sdSPI(VSPI);


String dataMessage;


void setup() {

  // Start serial communication for debugging purposes

  Serial.begin(115200);

 

  // Initialize SD card

  //SD.begin(SD_CS);  

  sdSPI.begin(SD_SCLK, SD_MISO, SD_MOSI, SD_CS);

  if(!SD.begin(SD_CS, sdSPI)) {

    Serial.println("Card Mount Failed");

    return;

  }

  Serial.println("1");

  uint8_t cardType = SD.cardType();

  if(cardType == CARD_NONE) {

    Serial.println("No SD card attached");

    return;

  }

  Serial.println("Initializing SD card...");

  if (!SD.begin(SD_CS)) {

    Serial.println("ERROR - SD card initialization failed!");

    return;    // init failed

  }

  Serial.println("2");

  // If the data.txt file doesn't exist

  // Create a file on the SD card and write the data labels

  File file = SD.open("/data1.txt");

  if(!file) {

    Serial.println("File doens't exist");

    Serial.println("Creating file...");

    writeFile(SD, "/data1.txt", "Reading ID, Date, Hour, Temperature \r\n");

  }

  else {

    Serial.println("File already exists");  

  }

  file.close();

  logSDCard();

  

}


void loop() {

  // The ESP32 will be in deep sleep

  // it never reaches the loop()

}


// Write the sensor readings on the SD card

void logSDCard() {

  //dataMessage = String(readingID) + "," + String(dayStamp) + "," + String(timeStamp) + "," + 

  //              String(temperature) + "\r\n";

  dataMessage = "Hello World \n";

  Serial.print("Save data: ");

  Serial.println(dataMessage);

  appendFile(SD, "/data1.txt", dataMessage.c_str());

}


// Write to the SD card (DON'T MODIFY THIS FUNCTION)

void writeFile(fs::FS &fs, const char * path, const char * message) {

  Serial.printf("Writing file: %s\n", path);


  File file = fs.open(path, FILE_WRITE);

  if(!file) {

    Serial.println("Failed to open file for writing");

    return;

  }

  if(file.print(message)) {

    Serial.println("File written");

  } else {

    Serial.println("Write failed");

  }

  file.close();

}


// Append data to the SD card (DON'T MODIFY THIS FUNCTION)

void appendFile(fs::FS &fs, const char * path, const char * message) {

  Serial.printf("Appending to file: %s\n", path);


  File file = fs.open(path, FILE_APPEND);

  if(!file) {

    Serial.println("Failed to open file for appending");

    return;

  }

  if(file.print(message)) {

    Serial.println("Message appended");

  } else {

    Serial.println("Append failed");

  }

  file.close();

}



https://randomnerdtutorials.com/esp32-spi-communication-arduino/.   esp32 cam spi pinouts



SPIMOSIMISOSCLKCS
VSPIGPIO 23GPIO 19GPIO 18GPIO 5
HSPIGPIO 13GPIO 12GPIO 14GPIO 15

 

Antenna Mod





flash with thony and micropython



a good write up for programming in arduino IDE








https://github.com/cflurin/xiaomi-mqtt.   Xiaomi Mqtt bridge gateway

https://gitlab.com/mike-sirs/xiaomi2mqtt


#products-powered-by-openmqttgatewaydecoder.      openmqttgateway library supported devices

https://github.com/pvvx/ATC_MiThermometer.   ATC bins.  a lot of info reading Xiaomi

https://github.com/custom-components/ble_monitor.   so called a passive monitor


Integrates BTHome BLE devices into Home Assistant.


https://github.com/pvvx/TLB2Z.     a low cost Zigbee receiving 3 BLU thermometers.

https://www.telink-semi.com/products/bluetooth-le/tlsr825x.    it sounds like the TL8258 is the candidate

ESP32-C3 TLSR8258 Zigbee Ultra Low Power IOT Development Board





https://github.com/devbis/z03mmc.      new Zigbee firmware for Xiaomi thermometer   


https://devbis.github.io/telink-zigbee/.      web interface for converting Xiaomi blu to Zigbee
https://popeen.com/tools/zigbee_flasher/.   popper its own custom firmware


















發佈者： 於 沒有留言:
訂閱： 留言 (Atom)